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dissemination efforts to promote 
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What GAO Found 

GAO as well as a congressionally mandated panel of experts, found that the 
WWC’s review process, which includes screening studies to determine if they 
meet WWC criteria, follows accepted standards for evaluating research on the 
effectiveness of education interventions. WWC is responding to 
recommendations made by the expert panel to further improve its review and 
reporting processes. For example, the panel recommended improvements in 
the way the WWC presents information to readers on the reasons why studies 
do not qualify for review. The WWC is revising a report template to include a 
table summarizing which studies met or did not meet WWC criteria for 
evaluating research. The WWC has also responded to researchers who have 
criticized the WWC for presenting limited information because its screening 
criteria exclude some rigorous research designs that may be appropriate for 
evaluating certain education programs, such as special education. The WWC 
responded to this criticism by creating new standards that include two 
additional study designs and by creating a new product, called a practice 
guide, which includes a wider range of research. 

WWC’s report output and scope increased under the current contract. For 
example, the WWC increased its production of various reports, introduced 
new products, and developed new processes for evaluating research. 

However, lES had a substantial backlog in its product review process from 
January 2009 to May 2010. The backlog generally decreased the timeliness of 
WWC reports, with 20 reports being delayed by up to 6 months. To support the 
increases in output and scope, WWC’s costs doubled from the previous 
contract to the current one. Both contracts designated about 60 percent of 
costs to production, while the other 40 percent of costs support other tasks, 
such as communications, dissemination, and process development. lES’ 
performance goals for the WWC primarily relate to the number of reports 
produced. However, lES has not developed performance measures related to 
the cost or usefulness of WWC products. 

Education uses WWC contractors. Regional Educational Laboratories (RELS) 
and the Doing What Works (DWW) Web site to disseminate information about 
WWC products; however, awareness and use of the WWC varies among states, 
districts, teachers, and principals. WWC contractors disseminate product 
information in various ways including email alerts and presentations. The 
RELs host events featuring WWC products for state, district, and school 
officials and DWW provides resources to educators based on WWC products. 
Based on our survey, officials from 33 of 38 state education agencies that 
responded to our survey and an estimated 42 percent of school districts have 
heard of the WWC. Those states and school districts generally used the WWC 
to a small or moderate extent to inform decisions on effective practices. 

Based on our survey, states and school districts reported that they would 
likely increase their use of the WWC if it included a broader array of 
information or more timely information. 
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GAO 

^^^^^^^—^Accountability * Integrity * Reliability 

United States Government Acconntability Office 
Washington, DC 20548 



July 23, 2010 

Congressional Committees 

The U.S. Department of Education’s What Works Clearinghouse (WWC or 
Clearinghouse) was established as a federal source of scientific evidence 
about “what works” in education. The Institute of Education Sciences 
(lES), a division of the U.S. Department of Education (Education), created 
the WWC in 2002, in part to help educators identify and use scientifically- 
based practices as specified in the Elementary and Secondary Education 
Act of 1965 (ESEA).^ The WWC, which is operated by an independent 
contractor, conducts systematic reviews of education research and 
disseminates information on its Web site about the effectiveness of the 
practices reported in these research studies. Currently operating under a 
$50 million 5-year contract, the Clearinghouse has generated criticism in 
the education research evaluation field on the timeliness of its reviews, its 
standards for study inclusion, and the methodological soundness of its 
research review process.^ 

An explanatory statement submitted in lieu of a conference report for the 
Omnibus Appropriations Act, 2009, directed GAO to examine how the 
WWC reviews education research and to address concerns about the 
operation, cost, and usefulness of the WWC.^ Specifically, GAO was 
required to determine whether the WWC review process met current 
standards for evaluating research and to examine the output and cost for 
completing reviews, the degree of consistency of review procedures 
across the various topics addressed, and the usefulness of the 
Clearinghouse for practicing educators. To conduct this work, we 
examined (1) the extent to which the WWC review process meets 
accepted standards for research evaluation and how the WWC has 
responded to recommendations and criticisms of its processes, (2) how 
the WWC’s output and costs have changed over time and how lES 



^20 U.S.C. §§ 6301-7941. The mission and functions to be performed by lES are set out at 20 
U.S.C. § 9611. 

^This is the second 6-year contract for the Clearinghouse. The first contract for about $27 
million expired in 2007. 

^H.R. Comm, on Appropriations, 111th Cong., Omnibus Appropriations Act, 2009: Comm. 
Print of the Corran. on Appropriations U.S. Representatives on H.R. 1105/Pubhc Law 111-8, 
at 1483 (Comm. Print 2009). 
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measures WWC performance, and (3) how WWC products are 
disseminated and how useful education professionals find them to be. 

To address all of our objectives, we interviewed and obtained information 
from lES officials and the current and former WWC contractors, as well as 
representatives from various educational organizations. In addition, to 
address objective 1, we reviewed a prior GAO report that examined WWC 
procedures and standards, an expert panel report that previously assessed 
the validity of the WWC review process, literature, and procedures used by 
other organizations that conduct systematic reviews of research. We also 
reviewed the Clearinghouse’s response to the expert panel and to specific 
criticisms in education research literature. To determine how performance 
and costs changed over time (objective 2), we analyzed the costs and 
productivity of the WWC contractors by reviewing budget, expenditure, 
and performance data. For objective 3, we administered a Web-based 
survey to state education agencies in all 50 states and the District of 
Columbia and a nationally representative sample of school districts;'' 
interviewed lES’s 10 Regional Educational Laboratories; and gathered 
nongeneralizeable information from teachers and principals at fom 
conferences. Appendix 1 explains our scope and methodology in more 
detail. We performed our work from September 2009 to July 2010 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 



Background 



The mission of the WWC is to be a central source of scientific evidence for 
what works in education.'* To accomplish this, the WWC reviews existing 
education research and posts information based on its reviews on the 
WWC Web site, http://ies.ed.gov/ncee/wwc/. 

The types of products currently available on the WWC Web site are 
described in table 1. 



■*Thirty-seven states and the District of Columbia and 74 percent of surveyed districts 
responded to our survey. 

^WWC’s mission is consistent with lES’s broader mission to bring rigorous and relevant 
research, evaluation, and statistics to the nation’s education system. 
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Table 1 : Products and Registries Available on the WWC Web Site 


Product 


Description 


Number 


Example 


Intervention reports 


Summarize all of the research reviewed for a 
particular intervention within a topic area. Each 
report offers an overview of the intervention, 
summarizes all relevant research, and provides a 
rating of effectiveness. Studies featured in 
intervention reports must meet WWC evidence 
standards with or without reservations. 


130 


Accelerated Reader: WWC reviewed the 
evidence pertaining to the effectiveness of 
this specific curriculum with respect to 
certain reading outcomes 


Practice guides 


Contain recommendations for educators to 
address challenges in their classrooms. Assign 
strength of evidence ratings to each 
recommendation (strong, moderate, low). Rely to 
some extent on expert opinion. 


12 


Structuring Out-of-School Time to Improve 
Academic Achievement: WWC published 
general recommendations on how to design 
out-of-school time programs that will 
increase student learning 


Quick reviews 


Assess the quality of research evidence from 
single studies recently featured in the media to 
determine if they meet WWC evidence standards. 


40 


Recess and Classroom Behavior: WWC 
reviewed a study profiled in the news that 
examined whether providing daily recess to 
third graders improves their classroom 
behavior 


Multimedia 


Audio files, video files, presentations, and 
transcripts from WWC events. 


N/A 


Reducing Behavior Problems in the 
Elementary School Classroom: WWC held 
a webinar featuring a practice guide on this 
topic 


Registry of evaluation 
researchers 


An online database of researchers who conduct 
evaluations of the effectiveness of educational 
interventions to help schools, school districts, and 
educational program developers Identify potential 
researchers. 


N/A 


Individual researchers and various 
organizations 


Registry of 
randomized 
controlled trials 


An online database of completed and in-progress 
randomized controlled trials in education. This 
resource is designed to help schools, school 
districts, and educational program developers 
Identify research regarding the effectiveness of 
educational interventions. 


N/A 


A Randomized Trial of Two Promising 
Interventions for Students with Attention 
Problems: WWC included this randomized 
controlled trial in its registry 



Source: GAO analysis of WWC information. 



Note: This table summarizes the WWC products and registries available as of May 18, 2010. The 
Clearinghouse previously published topic reports summarizing findings from all studies on all relevant 
interventions for a particular topic, such as beginning reading. These reports were replaced by 
dynamically generated summaries of evidence. 



In addition to the Clearinghouse, Education provides other technical 
assistance and research-related resowces to assist states, districts, and 
schools. Examples of research-related resomces include the Regional 
Educational Laboratories (REL) and the Doing What Works (DWW) Web 
site (http://dww.ed.gov): 
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• Regional Educational Laboratories. lES’s Regional Educational 
Laboratory Program is a network of 10 laboratories that conduct research 
and provide policymakers and practitioners with expert advice, training, 
and technical assistance on how to interpret findings from scientifically 
valid research.® 

• Doing What Works. Led by Education’s Office of Planning, Evaluation and 
Policy Development, DWW is a Web-based resource intended to help 
teachers, schools, districts, states and technical assistance providers 
implement research-based instructional practice. 



Initial Years of the What in 2002, lES awarded a $27 million 5-year contract to the initial contractors 

Works Clearinghouse to operate the Clearinghouse.^ The WWC contractors developed the 

Clearinghouse’s research review standards with lES and reviewed 
research related to topic areas considered to be pressing issues in 
education.® One of the goals of the Clearinghouse was to promote 
informed education decision making through a Web-based dissemination 
system that featured rigorous reviews of studies on the effectiveness of 
educational interventions. 

The WWC experienced a slow start due in part to the amount of work 
involved in developing a research review and reporting process that was 
valid, transparent, and replicable, according to the initial contractors. In 
developing the research review process, the contractors and lES 
addressed over 60 technical issues, such as determining what constitutes 
an acceptable level of participant loss (attrition) from a study and what 
methods should be in place to accommodate common education research 
techniques. In addition, initial plans for topic areas and reporting formats 
were modified. For example, lES decided to drop one planned topic area 



®The current priority for the 2006-2010 REL contract period is providing policymakers and 
practitioners with expert advice, training, and technical assistance on how to interpret the 
latest findings from scientifically valid research pertaining to requirements of the ESEA. 

Yhis contract was awarded to the American Institutes for Research and the Campbell 
Collaboration. In 2007, Education awarded the second 5-yeeir contract to Mathematica 
Policy Research, Inc. to operate the WWC. 

®The initial topic areas chosen in 2003 were: beginning reading, K-12 math achievement, 
dropout prevention, adult literacy, peer-assisted learning, reducing delinquency, and 
English language acquisition. 
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because lES officials determined it to be too broad.® The WWC and lES 
also spent a substantial amount of time developing and refining a reporting 
format to communicate research results to a lay audience. As a result, the 
WWC began releasing reports in 2006. By September 2007, the WWC had 
released 89 intervention reports, six topic reports, and three practice 
guides. 



WWC Research Review 
Process for Intervention 
Reports 



The WWC uses a three-step review process to assess the quality of studies 
and report on what the research indicates about the effectiveness of 
interventions. The WWC definition of interventions includes programs 
(such as whole school reform), products (such as a textbook or 
curriculum), practices (such as mixed-age grouping), or policies (such as 
class size reduction). “ The process begins with an initial screening of 
published and unpublished studies relevant to the intervention being 
reviewed. Studies are collected from electronic databases, journals, 
conference proceedings, and nominations solicited from the general 
public. The studies that pass initial screens are reviewed to determine 
whether they provide valid evidence of an intervention’s effectiveness. 
Using these studies, the WWC then synthesizes the evidence about the 
effectiveness of an intervention and publishes a report describing its 
findings. The Clearinghouse categorizes interventions as either having 
positive effects, potentially positive effects, mixed effects, no discernable 
effects, potentially negative effects, or negative effects (see fig. 1). 



®IES dropped the peer-assisted learning topic area, which would have covered 
interventions related to students learning with and from other students — generally in the 
same class and at a similar academic level. 

^°The WWC intervention reports primarily focus on branded products. 
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Figure 1 : WWC Research Review Process for interventions 



Step 1: Initial screening 



Step 2: Quality review 



Step 3: Synthesize evidence 



Is the study... 

• Randomized or 
quasi-experimental? 

• Published within 20 years 
of the beginning of the topic 
area review? 

• Focused on a relevant 
intervention to the topic 
under review? 

And does the study... 

• Target students in the topic 
area’s age or grade range 
and specified location? 

• Focus on populations 
relevant to the topic area 
(e.g., students with learning 
disabilities, English 
language learners)? 

• Report on at least 
one outcome relevant 
to the review? 



Studies 

meeting 

initiai 

screening 

criteria 



Two Ph.D.-level 
research analysts 
independently rate 
studies using a 
codebook that 
considers study 
design and 
execution, validity 
and reliability of 
outcome measures, 
and data analysis 
and reporting to 
evaluate the strength 
of the evidence in 
the study. 




Studies 
not meeting initiai 
screening criteria 




Studies 

meeting 

evidence 

standards 

(with or without 
reservations) 



Studies 
not meeting 
evidence 
standards 



Evidence from 
all studies 
meeting 
standards is 
synthesized and 
summarized for 
use in reports. 




Screened out 



Screened out 



Intervention 

reports 

summarize 
evidence on 
the effects 
of a specific 
intervention 



Source: GAO analysis of WWC guidelines. 



The WWC uses evidence standards to assess the strengths and weaknesses 
of a study’s methodology, such as the type of design it uses, the quality of 
the study’s data, and the appropriateness of the study’s statistical 
procedmes. Until recently, the WWC accepted two types of study 
designs — randomized experiments and quasi-experimental studies. “ Only 
randomized controlled trials (or randomized experiments) that WWC has 
determined to be well-designed and well-implemented are considered 
strong evidence and can receive the highest rating of “meets evidence 
standards without reservations.” The WWC also considers evidence from 
quasi-experiments it has determined to be well-designed and well- 



^*In June 2010 lES made public its standards for two additional study designs: regression 
discontinuity and single case design studies. 
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implemented. The highest rating a study using quasi-experimental design 
can receive is “meets evidence standards with reservations.” This rating 
category is intended to inform educators to interpret the study results with 
caution, as the results may reflect other factors, in addition to the impact 
of the intervention (see table 2). 



Table 2: Two Study Designs That Meet the WWC Standards with or without 
Reservations 


Study design 


Description 


Highest rating category 
WWC will assign if well- 
conducted, and why 


Randomized 

control-group 

experiments 


Compare the outcomes of 
groups that were randomly 
assigned either to the 
intervention group or to a 
nonparticipating control group 
before the intervention. Such an 
assignment helps ensure that 
any differences in outcomes 
can be attributed to the 
intervention. 


Meets evidence standards: 
Considers randomized 
experiments as the design that 
is most likely to yield unbiased 
estimates of a program’s 
impact on student outcomes. 


Comparison-group 

quasi-experiments 


Compare the outcomes of 
groups in which individuals are 
assigned to an intervention or 
control group in a way that 
minimizes observable 
differences between the groups 
that could affect outcomes. The 
researcher must demonstrate 
that the groups are equivalent 
on observable participant 
characteristics, such as age, 
grade level, prior academic 
achievement, or pretest results. 


Meets evidence standards with 
reservations: Even with 
equivalent observable 
characteristics, there may be 
differences in other participant 
characteristics related to the 
desired outcomes — for 
example, certain family or 
social structures that are 
unknown to the researcher. 


Source: GAO analysis of WWC information. 



lES Oversight and Support The WWC is administered by lES through a contract with a private 

ofthe WWC research organization. lES monitors implementation of the specific tasks 

detailed in the WWC contract by reviewing an armual work plan and 
monthly performance and expenditure reports submitted by the 
contractor. lES tracks implementation of the tasks, completion of 
performance goals, and adherence to the budget outlined in the contractor 
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work plan. The contractor monitors the work of any subcontractors that 
it uses to perform services such as research reviews, technological 
support, and communications support. 

lES is also involved in the development and dissemination of WWC 
products. lES reviews and approves proposed topics for WWC products, 
product formats, and the research review procedures. It also coordinates a 
group of independent researchers to peer review WWC products and 
reviews and approves all WWC products prior to public release. lES 
required the contractor to develop a communications plan to inform WWC 
customers about features of the Web site. 



WWC Reviews 
Research in 
Accordance with 
Accepted Standards 
and Has Responded to 
Recommendations 
and Criticisms 



WWC Follows Accepted 
Review Standards and Is 
Improving Its Review 
Process in Response to a 
Congressionally Mandated 
Expert Panel Report 



We found that the WWC review process follows generally accepted 
practices for reviewing research. Specifically, GAO’s November 2009 
report reviewing federally supported efforts to identify effective 
interventions found that the WWC determines whether a study provides 
credible evidence on effectiveness based on several dimensions, including 
the quality of the research design, how the study was implemented, and 
other technical considerations. “ Our 2009 report also noted that WWC 
follows a common approach to conducting its reviews, and provides 



^^lES uses some of this information to determine a performance-based award for the 
contractor. 

^^GAO, Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective 
Interventions, GAO-10-30 (Washington, D.C.: Nov. 23, 2009). 

^'^For example, WWC rates the credibility of a study’s evidence along a continuum. 
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information to help educators understand the body of existing research on 
specific interventions. 

Additionally, a congressionally mandated panel of experts found in 
October 2008 that WWC’s research review process was based on 
appropriate methods for judging the strength of the evidence regarding the 
effectiveness of interventions. For example, the panel agreed that the 
minimum qualifications a study must meet in order to be reviewed by the 
WWC are appropriate. The panel also found that WWC’s reporting process 
is reasonable and that the WWC provides succinct and relevant evidence 
on the effectiveness of education interventions. While the panel concluded 
that the WWC’s processes are generally appropriate, the panel made 
several recommendations to the WWC for continued improvement. The 
recommendations primarily related to establishing or clarifying 
procedures, reviewing statistical methods, and documenting the screening 
process. 



help educators understand the research behind a WWC report, the WWC (1) combines 
information on the size and number of studies reviewed to rate the extent of evidence as 
small or medium/large; (2) includes an overall rating of effectiveness on each measured 
outcome, which combines the size and direction of effects, statistical significance, and the 
quality of the research designs; and (3) reports the average improvement index across 
studies as the expected change in percentile rank for an average control group student if 
the student had received the intervention. 

Brown, D. Card, K. Dickersin, J. Greenhouse, J. Kling, and J. Littell, Expert Report on 
the What Works Clearinghouse, a report prepared by the National Board for Education 
Sciences, 2008. The expert panel was convened by the National Board for Education 
Sciences in 2008 in response to the Senate Appropriations Committee. S. Rep. No. 110-410 
at 228-29. The National Board for Education Sciences, consisting of 16 voting members 
appointed by the President with the advice and consent of the Senate, provides lES 
guidance and oversight. 20 U.S.C. § 9516. The mandate directed the Board to convene 
leading experts in rigorous evaluations to assess the WWC, specified that panel members 
should be free of conflicts of interest and that a report with any recommendations was to 
be submitted within 4 months. Expert panel members included economists, statisticians, 
and professors with expertise in other systematic review efforts in the fields of health care, 
and social policy. 
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The WWC implemented or is considering implementing 14 of the panel’s 17 
recommendations/^ The WWC implemented nine recommendations, in 
part by modifying some procedures and creating a procedures and 
standards handbook. For example, in response to the panel’s 
recommendation that the WWC include a table of study dispositions (e.g., 
whether studies meet WWC evidence standards) at the front of 
intervention reports, the WWC is modifying the report template to include 
a summary table along with the existing listing of dispositions in the 
reference section. The WWC also addressed panel concerns about 
technical issues in its review process by making its treatment of study 
attrition — ^the rate at which subjects drop out of a study — more consistent 
across topic areas. The panel noted that the WWC’s practice of 
determining acceptable attrition levels by topic area led to arbitrary 
inconsistencies across the topic areas. In response to the panel’s 
recommendation that the WWC reconsider this practice, the WWC took 
steps to increase its consistency by developing attrition guidance that 
applies to all topic areas. “ (See app. Ill for a table detailing the 
recommendations, WWC and lES’s response, and the status of any changes 
made in response to recommendations.) 

In addition, the WWC is considering implementing five other panel 
recommendations. For example, the panel raised concerns that the WWC 
does not document some potential conflicts of interest for the studies it 
reviews. In response to this concern, the WWC is considering tracking and 



WWC considered but is not implementing three of the panel’s recommendations. One 
recommendation suggested the WWC develop standards for documenting the program 
received by the comparison group that did not receive the intervention and potentially 
incorporating this information when making comparisons across studies and/or 
interventions. The other two related to WWC procedures for combining evidence across 
studies and asking study authors to reanalyze their data to correct a common error 
associated with the use of classrooms rather than individual students in data analysis. This 
mismatch can result in an overstatement of the statistical significance of the effects of the 
intervention. The WWC maintains that its current procedures are consistent with standard 
practices and has elected not to ask authors to reanalyze their data. See appendix III for 
more detail. 

^*The handbook documents the actions that WWC staff must take when reviewing research 
and the items that must be included in the reports, among other things. It is available at 
http://ies.ed.gov/ncee/wwc/pdf/wwc_procedures_v2_standards_handbook.pdf. 

^^WWC requires reviewers to select one of two levels of attrition (higher or lower) 
depending on the topic area and context. WWC allows a higher level of attrition for topic 
areas in which it assumes that attrition is due to factors that are not strongly related to the 
intervention. WWC allows a lower level of attrition for topic areas in which attrition may be 
due to certain individuals choosing not to participate in the intervention. 
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publishing whether studies of a program are funded or conducted by the 
program’s developers.^® Further, in response to the panel’s concern that 
the WWC’s screening process may exclude some eligible studies, the WWC 
is undertaking an evaluation of the reliability of its screening process. 
According to lES officials, they will postpone decisions about the 
recommendations until the newly appointed Commissioner for the WWC is 
on board and actively involved in the decision making. 



WWC Also Responded to 
Criticism That It Produces 
Limited and Potentially 
Misleading Information 



Some researchers claim that the WWC presents potentially misleading 
information by including brief experiments involving small numbers of 
students when evaluating interventions.^^ As a result, according to critics, 
educators may accept the WWC’s rating of the intervention’s effectiveness, 
even though the evidence behind the rating is limited. One researcher 
suggested the WWC emphasize larger studies that span significant periods 
of time and set a minimum sample size requirement. According to WWC 
staff, such changes would exclude valuable research and prevent the ’WWC 
from providing educators with research-based information about some 
interventions.^^ Instead of changing its treatment of sample size and study 
duration, the WWC began publishing information on the extent of the 
evidence supporting its findings in 2007. The WWC’s “extent of evidence” 
rating alerts educators when the WWC effectiveness ratings are based on a 
small amount of evidence. As figure 2 shows, 76 percent of interventions 
with positive or potentially positive ratings of effectiveness are based on a 
small amount of evidence (see fig. 2). 



^°In cases in which a study of a program is funded by the program developer, the study 
authors may have incentives to find positive effects of the program. Such incentives could 
call the validity of the study’s results into question. 

^Tor example, see Robert Slavin and Dewi Smith, “The Relationship Between Sample Sizes 
and Effect Sizes in Systematic Reviews in Education,” Educational Evaluation and Policy 
Analysis, vol. 31, no. 4 (2009): 500-506. 

^^WWC staff also contend that there is no statistical basis for setting a minimum sample 
size and doing so would arbitrarily ignore available evidence and potentially bias the 
findings of a systematic review. 
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Figure 2: Percentage of Interventions with Positive or Potentially Positive Ratings 
Categorized by the Amount of the Evidence Supporting Those Ratings 




Source: GAO analysis of WWC data. 



Small (51 interventions) 



Medium to large (13) 



Not rated (3) 



Note: The figure excludes seven interventions that were rated with different amounts of evidence as 
of April 27, 2010. Currently, the extent of evidence rating has two categories: small and medium to 
large. A rating of “medium to large” requires at least two studies and two schools across studies and 
a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is 
“small.” 



Further, researchers suggested that the WWC presents misleading 
information by rating interventions based on studies in which measures of 
student performance closely match the content taught to the intervention 
group, but not the control group. In such studies, higher test scores 
among the intervention group may not accurately represent the 
effectiveness of the intervention more generally. The researchers 
suggested that the WWC exclude such measures, or at least report on them 
separately. However, the WWC includes these measures because, 
according to lES officials, they answer questions about whether d iff erent 
interventions lead to different content knowledge. The WWC agrees that 
there is a concern regarding the reliability of outcome measures that are 
overly similar to the intervention, but maintains that WWC procedures 
attempt to exclude such measures. In addition, in response to researcher 



^^Robert E. Slavin and Nancy A. Madden, “Measures Inherent to Treatments in Program 
Effectiveness Reviews,” paper presented at the annual meetings of the Society for Research 
on Effective Education, Crystal City, Virginia, March 3-4, 2008; and Robert E. Slavin, “What 
Works? Issues in Synthesizing Educational Program Evaluations,” Educational Researcher, 
vol. 37, no. 1 (Jan./Feb. 2008): 5. 
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concerns that tests created by intervention developers may be biased,^'' the 
WWC added information to the intervention reports noting whether 
outcome measures are based on tests created by the developer. 

Some researchers and education professionals we interviewed suggested 
that the WWC produces limited information because its screening criteria 
are too restrictive — currently screening out about 90 percent of studies 
initially identified as potentially relevant (see fig. 3). Until recently, the 
WWC reviewed only two types of study designs — randomized experiments 
and quasi-experimental studies — and according to critics, this limited the 
amount and type of information available to educators. For example, 
staff from one REL noted that educators may not be able to find reviews of 
the interventions they are using or considering because so few studies 
meet WWC standards.^® Staff from another REL told us that if educators 
cannot find relevant and useful information, they may be discouraged from 
using evidence-based practices. Staff from a third lab noted that the 
narrow focus prevents educators from learning from less rigorous but 
nonetheless useful research, such as case studies describing an 
intervention’s costs and implementation requirements. 



^^Intervention developers may intentionally or unintentionally create a test that is more 
likely to favor the intervention because they have finamcial or other interests in the success 
of the intervention. 

^^While randomized control trials and quasi-experiments are considered to be rigorous 
approaches in assessing program effectiveness, they are not the only rigorous research 
designs available and may not always be appropriate. For example, such comparison group 
designs may not be appropriate for research on small numbers of students receiving special 
education services in a self-contained classroom. In such a Ccise, an in-depth case study 
may be more appropriate. Examples of other research methods include statistical analyses 
of observational data, such as student records, or analyses of surveys of an intervention’s 
participants. 

^^The WWC produces intervention reports noting when no studies meet stamdards. 
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Figure 3: Studies Reviewed That Meet WWC Evidence Standards 



Met evidence standards with or without reservations 



Initially reviewed 

2,669 studies 




92% Screened out 



226 studies 



Source: GAO analysis of WWC data. 



The WWC maintains that its screening criteria and study inclusion 
standards focus on studies that provide strong evidence of an 
intervention’s effectiveness, and lowering these standards could 
imdermine the validity of the findings reported by the WWC. Although the 
Clearinghouse screens out most studies, many of its reports have 
identified interventions with positive effects. Data from the contractor 
indicate that 58 percent of WWC’s intervention reports identify positive or 
potentially positive effects of interventions. While the WWC plans to 
continue using its methodological standards for reviewing randomized and 
quasi-experimental studies, the Clearinghouse acknowledges that the 
emphasis on randomized experiments and quasi-experiments can exclude 
useful information on interventions in certain topic areas, such as special 
education, that do not lend themselves to these study designs. The WWC 
created new standards to include additional study designs. 



^'^Specifically, the WWC developed standards — which were made pubhcly available in June 
2010 — for reviewing single-case and regression discontinuity designs. The WWC anticipates 
reviewing many studies with single-case designs — studies that involve repeated 
measurement of a single subject (e.g., a student or a classroom) — as it evaluates 
interventions for special education. Regression discontinuity designs compare outcomes 
for a treatment and control group that are formed based on the results of a preintervention 
measure. 
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The WWC also introduced practice guides in 2007 in response to criticisms 
that its intervention reviews exclude too much research and consequently 
provide limited information to educators. Written by a panel of experts, 
practice guides include recommendations for educators on various topics, 
such as reducing high school drop-out rates and reducing behavioral 
problems in the classroom. Whereas WWC’s intervention reviews are 
based entirely on studies that meet WWC evidence standards, practice 
guides also incorporate studies that do not have designs that are eligible 
for WWC review, or in some cases, are reviewed and do not meet WWC 
evidence standards, and include the views of experts. To develop 
recommendations, the practice guide panel reviews available literature 
about the particular topic and then meets several times to discuss the 
topic. Through consensus, the panel identifies effective practices based on 
the evidence. Once the practice guide is developed, it undergoes a quality 
assurance review by WWC and lES staff and external peer review. The 
following text box provides an example of practice guide 
recommendations and the level of evidence supporting them. 



^®The WWC has published 12 practice guides as of May 2010. The topics of these 12 practice 
guides are using data to support decision making, helping students navigate the path to 
college, structuring out-of-school time, assisting students in math, assisting students in 
reading, reducing behavior problems, dropout prevention, improving literacy, turning 
around low-performing schools, instruction for English language learners, encouraging 
girls in math and science, and organizing instruction and study. 
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Example of Practice Guide Recommendations and Evidence Leveis’ 

In 2009, the WWC published a practice guide to help educators assist students 
struggling with reading in the primary grades. The practice guide authors used an early 
detection and prevention framework known as Response to Intervention. The panel that 
authored the practice guide consisted of six researchers and one expert in 
implementation of the Response to Intervention model. Two WWC staff also assisted in 
the practice guide development. 

The panel’s recommendations follow. 


Recommendation 


Basis for recommendation 


Level of 
evidence" 


Screen all students for potential 
reading problems twice per year 
and monitor those with higher risk 


Numerous studies with designs 
that did not meet WWC evidence 
standards or that did not use 
samples that adequately 
resembled the population of 
interest 


Moderate 


Provide time for differentiated 
reading instruction for all students 
based on assessments of 
students’ current reading level 


One descriptive study and expert 
opinion 


Low 


Provide intensive, systematic 
instruction on foundational reading 
skills in small groups to students 
who score below the benchmark 
score 


1 1 studies that met WWC 
evidence standards 


Strong 


Monitor the progress of these 
students at least once a month 


3 studies that met WWC evidence 
standards, but did not evaluate the 
effectiveness of monitoring so no 
conclusive inferences could be 
made, and expert opinion 


Low 


Provide intensive interaction on a 
daily basis to students who show 
minimal progress after reasonable 
time in small group instruction 


5 studies that met WWC evidence 
standards but did not report 
statistically significant impacts on 
reading outcomes 


Low 



Source: GAO review of a WWC practice guide. 

"A strong rating indicates that studies supporting the recommendation generaiiy meet WWC standards. 
A moderate rating indicates that studies supporting the recommendation generaiiy meet WWC 
standards with reservations. A iow rating indicates the recommendation is based on expert opinion, 
derived from theory or experience, and supported with evidence that does not rise to the moderate or 
strong ieveis. 

"Our anaiysis of practice guide recommendations found that aimost haif of the 67 recommendations 
made in the 12 practice guides reieased as of May 2010 were based on a iow ievei of evidence. 
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WWC’s Output and 
Costs Increased; 
However, lES Has Not 
Developed Adequate 
Performance 
Measures Related to 
Cost or Product 
Usefulness 



WWC Increased Output WWC’s report output increased under the current contract, and its scope 
and Introduced New expanded to include new products and processes to support production. 

Products Under the current contract, the WWC increased its total number of 

publications from the first contract year to the second contract year and 
generally kept pace with its increased scope, as specified in the 
Clearinghouse’s annual plans. For example, the current contract calls for 
the WWC to increase the number of topic areas and intervention reports. 
Under the current contract, the WWC added three new topic areas and 
released 60 intervention reports, including 5 in the new topic areas as of 
June 2010.^“ In addition, the WWC produces practice guides and quick 
reviews and increased its production of both of these products between 
the first and second year of the current contract. Figure 4 shows the 
production of all three WWC products as of June 30, 2010, the end of the 
third contract year. 



^^lES requires the contractor to file WWC annual plans that outline planned product 
releeises and other deliverables. lES and the contractor update these plans once a year with 
revised estimates. 

^°Under the first contract (2002 to 2007), the WWC releeised 89 intervention reports, six 
topic reports, and three practice guides. 
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Figure 4: Publication Quantities, by Contract Year (CY) for Current WWC Contract 




Practice guides 

Contract 

year 

1 0 




Quick reviews 

Contract 
year 

1 



2 

3 



Source: GAO analysis of WWC data. 

Notes: CY is July 1 to June 30. CY1 covers this period for 2007 to 2008, CY2 for 2008 to 2009, and 
CY3 for 2009 to 2010. 

The WWC’s scope of work increased under the current contract with the 
addition of new products and work processes, as well as responsibilities 
related to the American Recovery and Reinvestment Act of 2009 (Recovery 
Act), which provided additional innovation and improvement funding.^^ 
The WWC is developing three new types of pubhcations and conducts an 
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Number of publications 



^'Pub. L. No. 11 1-5, 123 Stat. 116, 182. 
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annual review of Education-sponsored studies for lES’s internal use. 
Specifically, the WWC is developing research briefs, research perspectives, 
and practice briefs, which will focus on Education policy priorities. Like 
practice guides, the new publications will incorporate expert opinion and 
a broad range of research. Table 3 provides more information on these 
new initiatives.^^ 



Table 3: New WWC Publications and Reports 


Product 


Description 


Status 


Research briefs 


Short summaries of what research indicates 
about the effectiveness and implementation 
challenges of policies, practices, or issues in 
education. 


In process and 
template has 
been approved. 
First publications 
projected for 
release in 2010. 


Research 

perspectives 


Researchers’ perspectives on what research has 
found will work In addressing pressing 
educational issues. Topics will initially focus on 
issues relevant to the Recovery Act. 


In process and 
template has 
been approved. 
First publications 
projected for 
release in 2010. 


Practice briefs 


Provide explicit information on how to implement 
one practice from a WWC practice guide, and 
provides educators with research-based, how-to 
steps and strategies for overcoming roadblocks, 
and tools for educators. 


Template has 
been drafted but 
further work on 
this product is on 
hold pending 
direction from 
lES. 


Reviews of lES- 
sponsored 
studies (annual) 


lES uses this report to evaluate the research It 
funds. WWC reviews this research using WWC 
standards and reports on whether the research 
studies identify effective or promising practices. 


First produced in 
2008, with plans 
for annual 
reporting to lES. 


Source: GAO review of WWC contracts and annual plans. 



^^The current contract also requires WWC to create and maintain other resources on its 
Web site, such as registries of researchers and randomized trials and the WWC Policy and 
Procedures Handbook. lES noted that these deliverables are either new or significantly 
enhanced from those produced under the first WWC contract. 
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lES’s Reviews Have 
Delayed the Release of 
Some Reports 



While the WWC contractor increased its report production, lES’s review 
process did not keep pace with output. lES is responsible for 
administering independent peer reviews of all products and conducting 
final reviews and approvals before products are released, and has internal 
time frame estimates used in scheduling and completing such reviews. For 
example, according to lES planning documents, lES estimates 15 business 
days for the completion of peer reviews for intervention reports and 6 
business days for WWC quick reviews. However, throughout 2009, lES 
took increasingly more time to schedule and coordinate the completion of 
peer reviews for some intervention reports and quick reviews. As a result, 
the release of 20 reports — 11 intervention reports and nine quick 
reviews — was delayed by more than 6 months. For example, in the first 
quarter of the current contract year (third contract year, 2009 to 2010), lES 
took an average of over 50 business days to have intervention reports and 
quick reviews peer reviewed, compared to an average of 7 business days 
during the first quarter of second contract year (see fig. 5). 
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Figure 5: Average Time for lES Peer Reviews of Released Intervention Reports and Quick Reviews, by Contract Year (CY) and 
Quarter (Q) for Current WWC Contract 



Business days 

Intervention reports Quick reviews 




Source: GAO analysis of WWC data. 

Notes: CY is July 1 to June 30. CY1 covers this period for 2007 to 2008, CY2 for 2008 to 2009, and 
CY3 for 2009 to 2010. In CY3-Q2, no quick reviews were released. 

These delays in the lES-administered peer review process resulted in 
significant backlogs of intervention reports and quick reviews awaiting 
release. For example, as shown in figure 6, reports that entered the peer 
review process in the first quarter of the second contract year (CY2-Q1) 
were completed within that quarter. However, the majority of reports 
entering review the first quarter of the third contract year (CY3-Q1) 
remained in process for subsequent quarters. While the backlog persisted 
through the third quarter of the third contract year (CY3-Q3), the number 
of reports that completed peer review in the third and fourth quarters 
increased from prior quarters. Figure 6 shows that 11 intervention reports 
completed peer review in CY3-Q3 and an additional 27 completed peer 
review in CY3-Q4, compared with 4, 5, and 8 intervention reports in the 
prior three quarters. 
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Figure 6: lES Peer Review Backlog for Intervention Reports and Quick Reviews, by Contract Year (CY) and Quarter (Q) for 
Current WWC Contract 
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Source; GAO analysis of WWC data. 

Notes: CY is July 1 to June 30. CY1 covers this period for 2007 to 2008, CY2 for 2008 to 2009, and 
CY3 for 2009 to 2010. At the end of each quarter, any report remaining in peer review would carry 
over to the next quarter. For example, for intervention reports in CY2-Q3, 19 new reports entered 
peer review, joining the 1 report that remained from the previous quarter. Fourteen of these 20 
reports completed peer review, and 6 remained. 



lES attributed these delays to several factors and recently took steps to 
eliminate the backlog. lES officials told us that delays were, in part, 
attributable to difficulty in identifying and scheduling independent peer 
reviewers, vacancies in WWC-related positions at lES, and an increasing 
amount of research that met WWC standards. For example, lES officials 
told us identifying and scheduling a sufficient number of qualified, 
independent peer reviewers had become increasingly difficult because 



^^lES indicated that the amount of research available meeting WWC inclusion standards for 
a given report varied and had an impact on the number of staff hours required in the 
production of reports. Reports based on larger numbers of studies took more staff hours to 
complete than those based on less available evidence. 
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several former peer reviewers were now associated in some way with the 
WWC and therefore were no longer independent. To reduce the delays and 
eliminate the backlog, lES recently implemented a new database to help 
staff track and manage the work of peer reviewers and other WWC-related 
tasks. lES officials also told us that they began identifying additional 
potential peer reviewers using the WWC online registry of researchers. In 
addition, lES increased a staff member’s responsibilities related to 
scheduling and coordinating peer reviews. These efforts reduced the 
amount of time reports remain in the lES peer review process and 
eliminated the backlog as of June 2010.^“ 

In addition to delays in the peer review process, WWC contractors told us 
that many of their daily decisions need lES approval, and slow responses 
delayed contractor processes. For example, the contractor needs lES 
approval on the format and content of the products in development, 
hindering further work when responses are delayed. lES officials 
acknowledged that some delays in the approval process occurred during 
contract year three and told us that this was largely due to staff vacancies 
that they anticipate filling. 



The Cost of the Current 
WWC Contract Has 
Increased from the 
Previous One 



WWC’s contracted costs have doubled from about $5.3 million per year 
under the previous 5-year contract to the current level of about $10.7 
million per year.^^ The increase in contracted costs reflects the expanded 
scope — more publications and new products and processes — of the 
second contract compared to the first. lES’s contract for the WWC 
includes a variety of tasks that the contractor is responsible for, including 
tasks related to report production and product development. Table 4 
provides a description of six broad task categories and how they changed 
between contracts. 



^‘*We discussed the backlog and its causes with lES officials in February and May 2010. For 
the first six months of 2010, lES completed the review of 69 report products (intervention 
reports and quick reviews) — compared to 46 for the entirety of calendar year 2009 — thus 
eliminating the backlog. 

^^lES also spends about $200,000 per year on noncontracted WWC expenses — including 
internal salaries, independent peer review honorariums, and Web site support — which have 
not changed significantly between the two 5-year contracts. In addition, three practice 
guides were completed outside of the WWC contract, at a total cost of about $319,000. lES 
noted that these preliminary guides were produced through a less thorough process than 
the current process. 
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Table 4: Task Category Definitions and Changes between Contracts 


Task category 


Task category includes expenditures related 
to: 


Changes from first contract to current 
contract 


WWC products 


Conducting research reviews and developing and 
publishing WWC products 


New product types, expanded practice 

guide review process 

Stopped producing topic reports (2008) 


Strategic planning and 
coordination with lES 


Preparing annual plans, managing reporting 
requirements, and communications and workflow 
with lES 


New contractor database increased 
process documentation and reporting 
capabilities to lES 


Communications, collaboration, 
and dissemination of WWC 
products 


Maintaining WWC Help Desk 
Promoting WWC through various means 

Developing/implementing a 
communications/dissemination plan 


WWC staff attend conferences and 
coordinate some dissemination efforts 
with other lES departments in the current 
contract 


WWC development, process 
revisions, and maintenance 


Revising and developing review processes and 
policies 

Administrating and supporting technical staff 
training, technical advisory group, online registries, 
and conflict of interest procedures 


Enhanced review processes and 
standards, added new research designs 

Developed policy and procedures 
handbooks, new products, and staff 
training 


Web site and technical 
maintenance 


Coordinating content, maintaining 
databases/search functions, and processing 
federal data collection forms 


New online searchable system and 
database 


Award fees 


Fixed and performance-based contractor award 
fees based on a percentage of the overall contract 
total 


No change 


Source: GAO review of WWC contracts, annual plans, and budget documents. 



Our analyses of costs associated with these six broad task categories 
shows that the proportion of funds dedicated to producing WWC reports 
was about 60 percent under both contracts (see fig. 



^^Both contractors dedicated the same proportion of funds to WWC products. The first 
contractor primarily published products in the final year of the first contract (2007); 
however, products were produced, reviewed, and modified — but not published — ^prior to 
that year. As a result, despite limited publication in the first 4 years, a large portion of the 
first contractor’s expenditures were designated for direct product costs. 
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Figure 7: WWC Costs, by Task Categories and Contracts 



First 5-year contract ($26.5 million) 



Second 5-year contract ($53.3 million) 



Other costs 41% 



Other costs 41 % 



Practice 

guides 





Breakdown of other costs 

A Strategic planning and coordination with Education 
B Communications, collaboration, and dissemination 
C WWC development, process revisions, and staff training 
D Web site and technical maintenance 
E Award fees 



Source: GAO analysis of WWC budget data. 

Notes: Figures reflect actual expenditures for the first 5-year contract and updated budgeted 
expenditures for the current 5-year contract, which began on July 1 , 2007. Percentages do not add to 
100% due to rounding. Total WWC expenditures for the first 5-year contract were about $26,527,760, 
but cost category percentages do not reflect $1 ,222,71 4 billed by a co-contractor but not itemized by 
task. Category proportions for the first contract are estimates because IBS could not provide 
documentation that included final adjusted expenditures by tasks. The WWC budget is $53,315,166 
for the current 5-year contract, of which $23,643,891 had been spent as of October 31 , 2009. Cost 
category proportions for the current contract do not include $104,559 related to transition from the 
first contract to the second contract. 



The proportion of funds dedicated to some tasks changed from the first 
contract to the second. For example, costs for tasks related to process 
development and revisions doubled from 5 percent to 11 percent, 
supporting various activities such as expanding the practice guide review 
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process and revising the Clearinghouse’s procedures and standards 
handbook. According to lES officials, the current WWC contractor 
developed and implemented new or enhanced processes that affect all 
publications and deliverables. For example, the current contractor 
developed a standardized system for conducting and recording the WWC’s 
searches of research studies. 

Most WWC cost increases supported additional output and expansions in 
product scope. While under both contracts more resources were devoted 
to intervention reports than any other product, the proportion devoted to 
practice guides increased significantly, currently comprising about 21 
percent of total budgeted costs. lES noted that practice guides were only 
added during the last year of the prior contract, but are now a primary 
product. Other new WWC products make up a relatively small proportion 
of budgeted costs in the current contract, representing about 9 percent of 
the total contract budget combined. 



lES Has Not Developed 
Performance Measures 
Related to Production 
Costs or Product 
Usefulness 



lES established performance goals, which the WWC met or exceeded; 
however, these goals do not address production costs or the usefulness of 
WWC products. lES established WWC-related performance goals in its 
annual organizational assessment, but Education discontinued the use of 
these performance measures for fiscal year 2010.^® In addition, lES 
established performance goals for its WWC contractor in the contractor 
award fee plan, which lES uses to determine the amount of performance- 
based funds awarded to the contractor.’’’’ 



^According to WWC staff, this system allows them to use results from prior literature 
searches for related topics, rather than conducting new searches. The current contractor 
also designed and implemented standardized training for staff and subcontractors who 
evaluate research. All WWC reseeirch evaluators complete 2 days of training; are tested on 
WWC products, review standards, and policy; and have initial reviews with monitored 
before working independently. 

^*The Organizational Assessment — Education’s performance management system — was 
developed in response to the requirements of Executive Order 13460, Improving 
Government Program Performance, as well as the Office of Personnel Management’s 
requirement that each federal agency evaluate its principal offices on an annual basis. 

^^The contract award fee plan includes performance measures related to production, 
business management of the contract, and timeliness. Business management of the 
contract includes cost management, business relationships, efforts to meet small business 
subcontracting goals, and accurate billing. These measures are linked to work specified in 
the WWC contract and annual plans. 
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lES measured WWC program performance from fiscal year 2003 to fiscal 
year 2009, as part of Education’s Organizational Assessment — its 
departmentwide performance management system. The WWC-related 
performance goals included in Education’s Organizational Assessment 
focused on WWC Web site visits and the quantity of publications, both of 
which were areas of concern as the WWC was getting established. “ 
Specifically, these performance goals included increased WWC Web site 
visits, sustained productivity in the release of intervention reports and 
quick reviews, and increased practice guide production. The WWC met or 
exceeded these performance goals annually; however, according to lES 
officials, these performance goals will not be included in Education’s fiscal 
year 2010 Organizational Assessment, in part because lES is now satisfied 
with WWC activity in these areas.'*^ 

lES has not developed performance measures related to the cost of 
specific WWC products. lES officials noted that the costs per WWC 
publication vary greatly depending on the amount of available research 
relevant to the specifications of a product. For example, intervention 
reports based on a large number of studies meeting WWC standards take 
longer and cost more to produce than do reports for which few studies 
qualify for review. lES has tasked the current WWC contractor to develop 
ways to streamline production processes and to conduct a cost study, the 
results of which would improve budget estimates and strengthen lES’s 
monitoring of the contract. While the contractor has begun this work, lES 
officials told us that they do not know when cost-related performance 
measures, such as acceptable cost ranges for each type of product, will be 
established. 

WWC does not currently have a performance measure related to product 
usefulness. While Web site visits were tracked as a measure of WWC 
utilization in lES’s Organizational Assessment through fiscal year 2009, 
this metric did not assess the degree to which WWC products were 
reaching their target audience and did not provided any information on the 



^°While the WWC annually exceeded performance targets, it is difficult to interpret these 
results as the performance measures changed annually and, according to lES officials, the 
criteria for meeting them were negotiated well into the fiscal year. 

‘‘Tn addition, lES determined that the current and prior contractors generally met the 
award fee plan performance measures. 

^^The WWC’s award fee plan includes cost management components but has no cost per 
product measurements. 
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extent to which educators find WWC products to be useful. lES’s 2010 
budget justification calls for a representative survey of WWC use among 
education decision makers to be conducted by 2013. However, lES 
officials told us that they were unsure whether the survey would take 
place, and lES does not currently have a plan in place to implement this 
survey.'*^ 



Education Has Three 
Primary Ways to 
Disseminate 
Information about 
WWC Products, but 
Awareness and Use 
Vary among Target 
Audiences 



Education Has Various 
Ways to Disseminate 
Information about WWC 
Products, but Awareness 
of the Clearinghouse Is 
Generally Limited 



Education uses the WWC contractor, RELs, and DWW to disseminate 
information about the Clearinghouse to its target audience, which includes 
state and school district officials, as well as teachers and principals. In 
accordance with its contract, the WWC contractor disseminates 
information about its products electronically and through various events, 
such as formal presentations at conferences. The Clearinghouse’s 
electronic dissemination methods include an e-mail listserv, Web-based 
seminars (webinars), and newsletters. For example, the WWC sends out 
notices to its e-mail listserv, alerting subscribers of the availability of new 
products, including intervention reports, practice guides, and quick 
reviews. WWC staff told us that the webinars cover the same topics as 
their reports and are a relatively cost-effective way to disseminate 
information about products and methodology. In addition, WWC staff 



^^This survey would ascertain whether lES has met its goal that at least 25 percent of 
decision makers surveyed will have consulted the Clearinghouse prior to making decisions 
on interventions in reading, writing, mathematics, science, or teacher quality by 2013. 

^^WWC staff told us that the listserv had over 10,000 subscribers and that Web site visits 
increase after conferences. 
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disseminate information about WWC products at education conferences, 
such as teacher, principal, and researcher conferences. At these 
conferences, WWC staff may conduct formal presentations, have an 
exhibit featuring their products, or both. At conference exhibits. 
Clearinghouse staff answer questions about their products and provide 
literature to conference attendees. From July 2009 through June 2010, 
WWC staff were scheduled to present or have an exhibit at 14 conferences. 
WWC staff also told us that they work with other groups, such as 
education, research, and legislative organizations, in order to further 
disseminate information about WWC products to their members. 

In addition. Education disseminates information about WWC products 
through lES’s 10 RELs, which hold events that may feature information 
based on practice guides and refer educators to Clearinghouse products. 
Officials at all 10 RELs told us that they spent time disseminating 
information about WWC, in part by holding events that bridge research 
and practice. According to REL officials, these bridge events are attended 
primarily by school-, district-, and state-level education professionals and 
provide an opportunity for educators to discuss ways to implement 
research-based practices. Officials at all 10 RELs told us that bridge events 
focused on practice guides to some extent, and 7 indicated that WWC 
practice guides were the primary focus of these events. According to REL 
officials and WWC staff, these events sometimes included a WWC staff 
member to discuss methodology and panelists who helped develop the 
practice guides. RELs also disseminate research from WWC when 
responding to educator questions or concerns.'*'^ Officials from 7 of the 10 
RELs told us their respective RELs generally use relevant WWC products 
(practice guides and others) when searching for research-based 
information to address educator questions. 

In addition. Education’s Office of Planning, Evaluation, and Policy 
Development disseminates information about WWC practice guides on its 
DWW Web site, which provides an online library of resources designed to 
help educators implement research-based instructional practice. This Web 
site uses d iff erent formats to present content based primarily on WWC 
Practice Guides and provides examples of possible ways educators might 
apply WWC research findings. For instance, to help educators implement 



^^lES’s Web site hosts an “Ask A REL” page, where educators can submit questions. “Ask A 
REL” is described on the Web site as being a collaborative reference desk service provided 
by the 10 RELs that functions much in the same way as a technical reference library. 
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the recommendations from the practice guide on dropout prevention, the 
DWW Web site features slideshows with examples of supportive academic 
environments and interviews with educators and experts on dropout 
prevention. In addition, the Web site includes sample materials, such as 
lesson plan templates, that provide an example of how to implement 
recommendations. The DWW also includes information on the research 
behind the recommendations and a link to the WWC Web site and the 
individual practice guides. According to lES officials, a recent analysis of 
the DWW Web site traffic showed that 49 state Web sites have links to the 
DWW Web site, which helps disseminate WWC products further to the 
education community. 

We found that 33 of the 38 states'*® that responded to our survey reported 
that they had heard of the WWC. Based on our survey results, we estimate 
that 42 percent of school districts have heard of the WWC and that the 
percentage is greater for school districts that rely to a very large extent on 
external sources for information on research-based practices. 

School districts identified several sources of information about the 
Clearinghouse, including conferences and Education (see fig. 8). 



■*®Thirty-seven states and the District of Columbia responded to our survey. While the 
District of Columbia is not a state, we will refer to the survey respondents as representing 
38 states. 

'"^The 95 percent confidence interval for this estimate is (36.7, 46.6). 
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Figure 8: Sources from Which District Officials Heard of the WWC 

Estimated percentage of school districts 
90 



80 



72 73 




Potential sources of WWC information 



Not a source of WWC information 
Don’t know 

Source of WWC information 



Source: GAO analysis of school district survey responses. 

Notes: Estimates shown are based upon a probability survey. See appendix I for associated 
confidence intervals. The responses for other sources included state departments of education, 
Internet searches, and journals. 



While the majority of states have accessed the WWC Web site, we estimate 
that only 34 percent of school districts have done so. Specifically, among 
the states that responded to our survey, 33 of 38 states^® reported that they 
had accessed the WWC Web site at least once. In addition, 19 of states 
reported visiting the Web site at least seven times per year.^'* In contrast, 
an estimated 34 percent of school districts accessed the WWC Web site at 



^*While based on 38 state-level respondents, this analysis provides the minimum number 
(33) of states (overall) that have accessed the WWC Web site. Regardless of whether or not 
the 13 states that did not respond to our survey have accessed the Clearinghouse Web site, 
33 is about two-thirds of the 51 states and constitutes a majority of states. 

^°Our survey asked respondents to indicate the number of times they had accessed the 
WWC Web site. Answer choices included never; less than twice a year; between 2 and 6 
times per year; between 7 and 11 times per year; monthly; and more than once a month. 
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least once.™ Further, we estimate that only 11 percent of school districts 
visited the Web site at least seven times per year.®^ States and school 
districts that visited the WWC Web site less than seven times per year 
most often cited time constraints as the primary reason for their 
infrequent use.™ 

In addition to the WWC, states and school districts use a variety of other 
sources of information to identify effective education practices. Most 
states and school districts use several broad sources of information, such 
as academic journals, education periodicals, and associations of 
educators. For example, 37 states reported using academic journals to 
identify such practices, and we estimate that about 97 percent of school 
districts used academic journals.™ Overall, more school districts and 
states that responded to our survey used the WWC than used other 
research synthesis organizations.™ 

While the WWC also includes teachers and principals in its target 
audience, we found that relatively few of the teachers and principals we 
contacted at education conferences had heard of the WWC. While not a 
generalizeable sample, we found that out of a total of 391 teachers who 
completed our questiormaire at four education conferences, only 18 had 
accessed the WWC Web site.™ In addition, 341 teachers who had not 
accessed the WWC Web site told us they had not heard of the Web site. 
Similarly, among the 208 principals and other school administrators who 



™The 95 percent confidence interval for this estimate is (29.4 , 38.8). 

^^The 95 percent confidence interval for this estimate is (8.3, 14.8). 

^Fourteen states reported accessing the Clearinghouse six or fewer times a year, as did an 
estimated 72 percent of districts. In addition to time constraints (cited by 7 of the 14 
states), five states reported that they did not access the WWC more frequently because its 
content was not relevant to their decisions. 

™The 95 percent confidence interval for this estimate is (94.2, 98.4). 

™Our survey asked states and districts to report how useful certain sources of information 
were in identifying effective education practices. The sources ranged from general 
(personal experience, education periodicals) to specific (RELs, federal outreach centers). 
In addition, we listed several research synthesis organizations by name, including the WWC 
and the Best Evidence Encyclopedia. See appendix II for more details. 

^Between November 2009 and February 2010, we attended a regional conference for the 
National Council for Teachers of Mathematics in Nashville, Tenn., as well as three national 
conferences: National Council for Teachers of English, ASCD (formerly the Association of 
Supervision and Curriculum Development), and the National Association of Secondary 
School Principals. 
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completed the questionnaire, only 32 had accessed the WWC Web site. 
Further, 135 principals and other school administrators told us they had 
not heard of the WWC. 



States and School Districts 
Generally Used the 
Clearinghouse to a Small 
or Moderate Extent to 
Inform Decisions and Used 
Specific WWC Products to 
Varying Extents 



Based on our survey, most states and school districts that reported 
accessing the WWC Web site used it to inform decisions on effective 
education practices — a stated purpose of the WWC — ^to a small or 
moderate extent. Specifically, 25 of the 33 states that use the 
Clearinghouse indicated that they use it to a small or moderate extent to 
inform their decisions, while 6 reported using it to a large or very large 
extent. “ We estimate that 72 percent of school districts that have accessed 
the Clearinghouse use the WWC to inform education decisions to a small 
or moderate extent, while only 18 percent use it to a large or very large 
extent. 



States that used the WWC to inform decisions reported that they used the 
Clearinghouse for various purposes, including informing professional 
development and curriculum decisions. For example, 25 states reported 
using the Clearinghouse to inform professional development programs for 
teachers, and 22 reported using it to inform curriculum decisions. Fewer 
states used the Clearinghouse to advise districts that were not making 
adequate yearly progress (AYP) in meeting academic targets or to develop 
improvement plans for such districts. (Fig. 9 provides a breakdown of the 
extent to which these states use the Clearinghouse for various purposes.) 



^®One state official responded that he and his staff had not used the WWC to inform any 
decisions on effective education practices, while another state official responded “Don’t 
know.” 

^Vhe 95 percent confidence intervals for these estimates are (55.5, 87.7) and (10.3, 29.8), 
respectively. 
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Figure 9: Extent to Which States Use WWC for Various Purposes 



Number of states 
15 



12 12 










// 






Not at all 
Small extent 
Moderate extent 
Large or very large extent 



Source: GAO analysis of state responses. 

Note: “I Don’t Know” was also a response option, and is not dispiayed in the figure. 



In addition, we estimate that among school districts that use the WWC to 
inform decisions on effective education practices, about 90 percent used it 
to inform curriculum decisions at least to a small extent, similar to the 
percentage that used the WWC to inform professional development 
decisions. However, fewer school districts used it to advise schools that 
did not meet academic goals or to develop school-level plans to help such 
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schools improve/** Figure 10 provides a breakdown of the extent to which 
these school districts use the Clearinghouse for various purposes. 



Figure 10: Extent to Which School Districts That Have Used the Clearinghouse 
Used It for Various Purposes 



Estimated percentage of school districts 







Not at all 
Small extent 
Moderate extent 
Large or very large extent 



Source: GAO analysis of school district survey responses. 

Note: Estimates shown are based upon a probability survey. See appendix I for associated 
confidence intervals. “I Don’t Know” was also a response option, and is not displayed in the figure. 



^®The estimates and their 96 percent confidence intervals were as follows: inform 
curriculum decisions — 93 percent (86.6, 97.3); inform professional development of 
teachers — 89.4 percent (81.7, 94.7). 
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States reported using specific WWC products — intervention reports and 
practice guides — ^more than quick reviews. Specifically, of the states that 
had used the Clearinghouse, 21 reported that they used intervention 
reports and 20 reported using practice guides, while only 12 reported using 
quick reviews. States used intervention reports and practice guides to a 
similar extent to inform education decisions. For example, for each 
product, six states reported using them to large or very large extent to 
inform such decisions (see fig. 11). 



Figure 11: Extent to Which States Use Specific WWC Products 

Number of states 
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Practice guides 



Quick reviews 



Not at all 
Small extent 
Moderate extent 
Large or very large extent 



Source: GAO analysis of state survey responses. 



However, the relative use of specific WWC products was different among 
school districts. We estimate that among school districts that use the 
Clearinghouse to inform decisions on effective education practices, more 
school districts use intervention reports relative to practice guides or 
quick reviews. Specifically, we estimate that 74 percent of those school 
districts that use the WWC have used its intervention reports to inform 
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education decisions/'* while practice guides and quick reviews were each 
used by about half of such districts.®" Based on our survey, an estimated 21 
percent of school districts that use the WWC have used intervention 
reports to a large or very large extent,"* while about 10 percent use 
practice guides to a large or very large extent (see fig. 12).® 



Figure 12: Extent of Specific Product Use among Districts That Use the 
Clearinghouse 

Estimated percentage of school districts 
50 



42 




Not at all 
Small extent 
Moderate extent 
Large or very large extent 



Source: GAO analysis of school district survey responses. 

Note: Estimates shown are based upon a probability survey. See appendix I for associated 
confidence intervals. 



""The 95 percent confidence interval for this estimate is (64.3, 82.8). 

"°The estimates and their 96 percent confidence intervals were as follows: practice 
guides — 64.6 percent (44.4, 64.8); quick reviews — 54.2 percent (44.4, 64.3). 

"*The 95 percent confidence interval for this estimate is (13.7, 30). 

"^The 95 percent confidence interval for this estimate is (6.3, 17). 
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States and School Districts 
Would Likely Increase 
Their Use of the 
Clearinghouse If the WWC 
Made Certain Changes 



Many states and school districts that had accessed the Clearinghouse 
reported that they would likely increase their use of the WWC if the 
Clearinghouse provided a broader array of information. For example, 
many states and school districts would be likely to increase their use of 
the Clearinghouse if it reviewed more studies, covered additional topics, 
or provided more relevant or timely reports. For example, 21 of the 33 
states that had used the Clearinghouse reported that they would be 
somewhat or very likely to use the Clearinghouse more often if it had 
reviews that were more timely (see fig. 13).'® 



^^Twenty-six states that had used the Clearinghouse reported that they would be somewhat 
likely or very likely to increase their use of the WWC if it had a greater number of 
intervention reports showing positive effects — a number which depends both on the 
number of interventions that the WWC reviews and whether the results of available 
research meeting WWC standards show positive effects. 
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Figure 13: Number of States That Reported They Would Likely Increase Their Use of 
WWC Given Certain Changes 
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Source: GAO analysis of state survey responses. 



In addition, based on our survey, we estimate that about two thirds of 
school districts that had accessed the Clearinghouse would likely increase 
their use if it included reviews of programs or interventions being used or 
considered in their school district. “ An estimated 50 percent of school 
districts would hkely increase their use of the Clearinghouse if it had 
reviews that were more timely (see fig. 14).®^ 



^‘'The 95 percent confidence interval for this estimate is (69.8, 76.1). 
^^The 95 percent confidence interval for this estimate is (41.3, 58.6). 
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Figure 14: Estimated Percent of School Districts That Have Accessed the WWC 
That Would Likely Increase Their Use of the WWC Given Various Changes 
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Source: GAO analysis of school district survey responses. 

Note: Estimates shown are based upon a probability survey. See appendix I for associated 
confidence intervals. 



Conclusions 



In 2007, Education substantially increased its financial investment in the 
WWC, and the Clearinghouse is significantly expanding its scope in an 
effort to better serve its target audience. Some of the new products aim to 
be more responsive to educators and education decision makers by 
providing timely information about evidence-based practices relevant to 
pressing needs. Such information could help states and districts identify 
strategies as they implement educational reform efforts — such as 
reforming low-performing schools or improving professional 
development — under ESEA and the Recovery Act. For example, WWC 
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research perspectives, still in development, are intended to help education 
decision makers as they address challenges related to spending Recovery 
Act funds. However, the development of these products and the release of 
other products were delayed, in part, by a substantial backlog in IBS’s 
review and approval processes. These delays hindered the timely release 
of several publications, and some products were released months after 
they were completed by the contractor. While lES recently eliminated the 
backlog, educators need to be able to rely on the Clearinghouse for timely 
and relevant information. According to our survey, many states and school 
districts reported that they would likely increase their use of the 
Clearinghouse if it released information more quickly. 

While lES has increased annual report production, lES has not established 
reasonable production cost ranges or specific cost-related performance 
measures related to each product type. Without acceptable per product 
cost ranges, it is difficult for lES to assess the reasonableness of costs 
associated with certain products, even as lES takes steps to streamline 
production. IBS’s current study on costs may help lES establish acceptable 
cost ranges that could inform IBS’s performance measurements related to 
the WWC. In addition, such information could inform cost comparisons 
between the WWC and other research evaluation organizations or provide 
baselines for future contractor work. 

In addition, IBS has not established meaningful performance measures 
related to product usefulness. Until fiscal year 2010, IBS tracked visits to 
its Web site and annual report production as a way to measure the 
productivity of the Clearinghouse. While these measures were important 
to accurately track the ’WWC’s initial growth, they did not evaluate the 
degree to which the products were meeting the needs of educators. 
Specifically, IBS currently does not have a way to gauge user satisfaction 
with WWC products, which is a common practice when developing and 
providing new products. Further, while IBS currently incorporates some 
feedback from the WWC Web site users, to inform future topic areas, it 
does not systematically gauge its target audience’s major areas of interest 
or concern — such as gathering information on interventions currently 
being used or considered in specific school districts or states. IBS decides 
how to spread its limited resources across the various product types 
without directly measuring the extent to which educators use the WWC or 
how useful they find the various products to be. Measuring the use and 
usefulness of its products could help IBS continue to improve content, 
develop products, and respond to the needs of educators and 
policymakers. 
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While some educators and policymakers find WWC products useful, many 
other educators are not familiar with the Clearinghouse. lES has spent a 
substantial amount of money, time, and effort producing various 
summaries of evidence-based practices, which cover both specific 
education interventions and general practices. This investment in the 
WWC was made in order to inform education professionals at all levels — 
from classroom teachers to policymakers — as they make decisions on how 
best to educate the nation’s children. Improved dissemination of WWC 
products could increase awareness and use of the WWC. Increased use of 
the Clearinghouse could help education professionals identify and 
implement effective educational interventions and practices, and 
potentially lead to increased student achievement. 



R6COrn.m.6ndcltionS for making the following four recommendations based on our review. 

Executive Action To consistently release WWC products in a timely manner, we recommend 

the Secretary of Education direct lES to develop and implement strategies 
that help avoid future backlogs and ensure that lES’s review and approval 
processes keep pace with increased contractor production. Strategies 
could include shifting lES resources to ensure sufficient staff time for 
managing the peer review process and streamlining its approval processes. 



To better track the costs and usefulness of the WWC, we recommend that 
the Secretary of Education direct lES to 



• incorporate findings from its cost studies to develop performance 

measures related to costs, such as identifying a range of acceptable costs 
per product and using that information to monitor contractor spending; 
and 



• develop performance measures related to product usefulness and 

periodically assess whether WWC products are meeting the needs of target 
audiences by gathering information on product usefulness in the proposed 
survey or through other means. 

To reach more members of the target audience, we recommend the 
Secretary of Education direct lES to assess and improve its dissemination 
efforts to promote greater awareness and use of the WWC, for example, by 
developing a way to inform school districts of new products or 
encouraging educator professional development programs to focus on 
research-based practices such as those discussed in practice guides. 
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Agency Comments 
and Our Evaluation 


We provided a draft of this report to the U.S. Department of Education for 
review and comment. Education officials provided written comments on a 
draft of this report, which are reproduced in appendix IV. Education also 
provided technical comments, which we incorporated into the report as 
appropriate. 

Education generally agreed with our recommendations. Specifically, 
Education agreed to om recommendations on consistently releasing WWC 
products in a timely marmer and assessing and improving its dissemination 
efforts. In its response to our recommendation on tracking the cost and 
usefulness of the WWC and its products. Education noted that lES has 
taken some steps that address the recommendation. With regard to costs. 
Education stated that it intends to incorporate the results of current cost 
studies into future work plans and monitoring efforts. We continue to 
recommend that these results be used to irrform performance measures 
related to costs for future operations. With regard to tracking the 
usefulness of the WWC, Education noted that it uses a variety of tools to 
gather consumer input, such as a Help Desk and online voting for future 
report topics. While such feedback provides some irrformation to the 
WWC, it relies on existing users and reflects the views of those users who 
provide feedback, rather than those of the broader population. However, 
as shown in our survey, only an estimated 34 percent of school districts 
have accessed the WWC Web site at least once — and fewer have used the 
Web site frequently. Education also noted that it would include a customer 
satisfaction survey in lES’s review of its own performance, but whether 
the survey would be directed at current Clearinghouse customers or a 
broader audience, or whether the survey would identify how useful 
various WWC products are and how the WWC can be improved is unclear. 
More nationally representative irrformation could help lES prioritize topics 
for intervention reports and practice guides and inform budget priorities. 




We are sending copies of this report to the appropriate congressional 
committees, the Secretary of the U.S. Department of Education, and other 
interested parties. In addition, the report will be available at no charge on 
the GAO Web site at http://www.gao.gov. 
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If you or your staffs have any questions regarding this report, please 
contact me at (202) 512-7215 or ashbyc@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on the 
last page of this report. GAO staff who made major contributors to this 
report are listed in appendix V. 

Cornelia M. Ashby 

Director, Education, Workforce, 

and Income Security Issues 
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The Honorable Tom Harkin 
Chairman 

The Honorable Thad Cochran 
Ranking Member 

Subcommittee on Labor, Health and 
Human Services, Education and Related Agencies 
Committee on Appropriations 
United States Senate 

The Honorable David Obey 
Chairman 

The Honorable Todd Tiahrt 
Ranking Member 

Subcommittee on Labor, Health and 
Human Services, Education and Related Agencies 
Committee on Appropriations 
House of Representatives 
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Appendix I; Objectives, Scope, and 
Methodology 



To address all three objectives, we interviewed officials from the Institute 
of Education Sciences (lES), What Works Clearinghouse (WWC or 
Clearinghouse) contractors, and representatives from various educational 
organizations. To assess the research review process used by the lES’s 
WWC, we reviewed WWC standards and procedures, reviewed an expert 
panel report that assessed the validity of the WWC review process, and 
collected information about the extent to which the WWC has 
implemented the panel’s recommendations. To determine how 
performance and costs changed over time, we analyzed the costs and 
productivity of the two WWC contractors. To obtain information about the 
usefulness of WWC products, we conducted a Web-based survey of all 
state education agencies and a nationally representative sample of school 
districts. We also collected information about the usefulness of the WWC 
from teachers and principals at four education conferences. We conducted 
our work from September 2009 through July 2010 in accordance with 
generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, appropriate 
evidence to provide a reasonable basis for om: findings and conclusions 
based on our audit objectives. We believe that the evidence obtained 
provides a reasonable basis for om: findings and conclusions based on our 
audit objectives. 



Assessment of WWC GAO previously assessed the procedures and criteria used by the WWC by 

Research Review Process reviewing documents and interviewing lES officials and WWC 

contractors.^ We reviewed WWC standards and procedures and examined 
the degree of consistency of these standards and procedures across 
education topic areas. We also reviewed the findings and 
recommendations from an expert panel report that assessed the validity of 
the WWC review process.^ We obtained information from lES officials and 
WWC contractors on the extent to which the WWC has implemented the 
panel’s recommendations. Further, we identified other concerns about the 



^GAO, Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective 
Interventions, GAO-10-30 (Washington, D.C.: Nov. 23, 2009). 

^The expert panel was convened by the National Board for Education Sciences in 2008 in 
response to a mandate from the Senate Appropriations Committee. The National Board for 
Education Sciences, consisting of 16 voting members appointed by the President with the 
advice and consent of the Senate, provides lES guidance and oversight. The mandate 
directed the board to convene leading experts in rigorous evaluations to eissess the WWC 
and specified that panel members should be free of conflicts of interest. Expert panel 
members included economists, statisticians, and professors with expertise in other 
systematic review efforts in the fields of health care, social pohcy, and education. 
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Appendix I: Objectives, Scope, and 
Methodology 



WWC review process through a literature review and interviews with 
researchers, and we interviewed lES officials and WWC contractors to 
assess the extent to which the Clearinghouse has addressed these 
concerns. We also examined the degree to which the WWC’s review 
process is similar to that used by other entities engaged in systematic 
research review efforts. 



Performance and Cost 
Data Analyses 



To determine how performance and costs changed over time, we analyzed 
the costs and productivity of the two WWC contractors. We reviewed 
budget data and product release dates to analyze cost and productivity 
trends of the WWC. To examine performance, we interviewed the two 
contractors, as well as lES officials, and compared lES’s performance 
measures and goals to actual outcomes. 

We assessed the reliability of the WWC performance and cost data by (1) 
reviewing existing information about the data and the system that 
produced them and (2) interviewing agency officials knowledgeable about 
the data. We determined that the data were sufficiently reliable for the 
purposes of this report. 



Survey of States and To determine how WWC products are disseminated, we interviewed 

School Districts officials from lES and all 10 RELs, as well as WWC contractors. To 

determine how useful education professionals find WWC products to be, 
we designed and administered a Web-based survey of state education 
agencies in the 50 states and the District of Columbia and a nationally 
representative sample of local educational agencies (LEA). Specifically, 
the survey asked officials about (1) their general sources of information on 
effective educational practices, (2) the extent to which they use WWC 
products to inform curriculum decisions (including questions on specific 
intervention reports and practice guides), (3) how useful the officials find 
the information in the WWC, (4) the likelihood they would increase their 
usage if certain changes were made to the WWC Web site, and (5) the 
extent to which the officials use the Doing What Works and Best Evidence 
Encyclopedia Web sites to inform curriculum decisions and how useful 
the officials find these other information sources to be. We reproduce the 
questions we used in our analysis in figure 15. The survey was 
administered from February 18, 2010 to April 14, 2010. 

To determine how the WWC was being used at the state level, we surveyed 
the state Secretary, Commissioner, or Superintendent of Education in the 
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50 states and the District of Columbia. Out of the 51 state officials 
surveyed, 38 responded to the survey. 

To determine how the WWC was being used at the school district level, we 
surveyed a nationally representative sample of school districts across the 
country. We selected a stratified random sample of 625 LEAs from the 
population of 17,620 LEAs included in our sample frame of data obtained 
from the Common Core of Data for the 2007-08 school year. A total of 454 
LEAs responded, resulting in a final response rate of 74 percent. Because 
we surveyed a sample of LEAs, survey results for the district are estimates 
of a population of LEAs and thus are subject to sampling errors that are 
associated with samples of this size and type. Our sample is only one of a 
large number of samples that we might have drawn. As each sample could 
have provided d iff erent estimates, we express our confidence in the 
precision of our particular sample’s results as a 95 percent confidence 
interval (e.g., plus or minus 10 percentage points). We excluded 12 of the 
sampled LEAs for various reasons — 6 were closed, 3 did not administer 
any schools, 2 managed schools in a correctional facility, and 1 was a 
private school — and therefore were considered out of scope. All estimates 
produced from the sample and presented in this report are representative 
of the in-scope population. 

The practical difficulties of conducting any survey may introduce 
nonsampling errors, such as difficulties interpreting a particular question, 
which can introduce unwanted variability into the survey results. We took 
steps to minimize nonsampling errors by pretesting the questionnaire over 
the phone with officials from two school districts and one state 
department of education in November and December 2009. We conducted 
pretests to verify that (1) the questions were clear and unambiguous, (2) 
terminology was used correctly, (3) the questionnaire did not place an 
undue burden on officials, and (4) the questionnaire was comprehensive 
and unbiased. An independent reviewer within GAO also reviewed a draft 
of the questionnaire prior to its administration. We made revisions to the 
questiormaire based on feedback from the pretests and independent 
review before administering the survey. 

The survey-related data used in this report is based on the state and school 
district responses to the survey questions. 
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Figure 15: GAO’s Web-based Survey of State Departments of Education and Local Educational Agencies in the 50 States and 
the District of Columbia 



3. To what extent, if at all, does your rely on external 

evidence-based research to Inform curriculum decisions? 

(Check only one answer) 

1 . o Very large extent 

2. O Large extent 

3. O Moderate extent 

4. O Small extent 

5. O Does not use external evidence-based research 

6. O Don't know 



4. How useful. If at all, are each of the following research evaluation 
resources to you or your staff in identifying effective practices to 
implement in your ? 



(Please choose one response for each resource.) 





Very 

useful 


Useful 


Somewhat 

useful 


Slightly 

useful 


Not at all 
useful 


No 

opinion 


Have not 
used this 
source of 
information 


4a. Best Evidence Encyclopedia (Johns Hopkins 
University) 


O 


O 


O 


O 


O 


o 


O 


4b. Child Trends 


O 


O 


o 


o 


O 


o 


O 


4c. Coalition for Evidence Based Policy 


O 


o 


o 


o 


O 


o 


o 


4d. Doing What Works 


O 


o 


o 


o 


O 


o 


o 


4e. RAND's Promising Practices 


O 


o 


o 


o 


O 


o 


o 


4f. What Works Clearinghouse 


O 


o 


o 


o 


O 


o 


o 


4g. Other research synthesis clearinghouses 


O 


o 


o 


o 


O 


o 


o 
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5. Recognizing that research evaluation resources are not necessariiy 
the primary sources of information used to identify effective 
education practices, GAO is aiso interested in the roie of other 
information sources. 

How usefui, if at ali, are each of the foiiowing sources to you or your 
staff in identifying effective practices to impiement in your ? 

(Please choose one response for each resource.) 

Have not 
used this 





Very 

useful 


Useful 


Somewhat 

useful 


Slightly 

useful 


Not at all 
useful 


No 

opinion 


source of 
information 


5a. Academic journals 


O 


o 


O 


O 


O 


o 


O 


5b. Education-related periodicals 


O 


o 


o 


o 


O 


o 


O 


5c. Online databases (ERIC or others) 


o 


o 


o 


o 


O 


o 


o 


5d. University-based research institutions 


o 


o 


o 


o 


O 


o 


o 


5e. Non-profit organizations 


o 


o 


o 


o 


O 


o 


o 


5f. Associations of educators or researchers 


o 


o 


o 


o 


O 


o 


o 


5g. Peer conferences 


o 


o 


o 


o 


O 


o 


o 


5h. Regional Education Laboratories 
(Department of Ed) 


o 


o 


o 


o 


O 


o 


o 


5i. Other federal outreach centers 


o 


o 


o 


o 


O 


o 


o 


5j. State government offices and/or outreach 
centers 


o 


o 


o 


o 


O 


o 


o 


5k. Local data and/or internal research 


o 


o 


o 


o 


O 


o 


o 


51. Community and parent input 


o 


o 


o 


o 


O 


o 


o 


5m. Mentors/Colleagues 


o 


o 


o 


o 


O 


o 


o 


5n. Personal experience 


o 


o 


o 


o 


O 


o 


o 


5o. Other resource (Please specify below) 


o 


o 


o 


o 


O 


o 


o 
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Other resource: 



7. Have you or your staff heard of the What Works Clearinghouse 
(WWC)? 

(Check only one answer) 

1 . o Yes 

2. O No (Go to Section 3: Use of Doing What Works) 

3. O Don't know (Go to Section 3: Use of Doing What Works) 

8. From which of the following source(s) did you or your staff hear 
about the WWC? 

(Please choose one response for each source.) 

Yes No Don't know 



8a. Conferences 


o 


o 


o 


8b. Peers 


o 


o 


o 


8c. Regionai Education Labs 


o 


o 


o 


8d. U.S. Deparment of Education 


o 


o 


o 


8e. Other source (Please specify below) 


o 


o 


o 



Other source: 



9. How frequently, if at all, do you or your staff access the WWC 
website? 

(Check only one answer) 

1. o Never (Go to question 28.) 

2. O Less than twice a year (Go to question 10.) 

3. O Between 2 and 6 times per year (Go to question 10.) 

4. o Between 7 and 1 1 times per year (Go to question 11.) 

5. o Monthly (Go to question 11.) 

6. o More than once a month (Go to question 11.) 
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10. You indicated that you or your staff access the WWC website iess 
than 7 times per year. Which of the foiiowing reasons best describes 
why you and your staff do not access the website more frequentiy? 

(Check only one answer) 

1. o Time Constraints 

2. o Content is not relevant to our decisions 

3. O Disagree with recommendations on the site 

4. o Site is difficult to navigate 

5. o Other reason (Please specify below) 

Other reason: 



11 . To what extent, if at aii, have you or your staff used the WWC 
website to inform decisions on effective education practices? 

(Check only one answer) 



1 . o Very large extent (Go to question 13.) 

2. o Large extent (Go to question 13.) 

3. O Moderate extent (Go to question 13.) 

4. o Small extent (Go to question 13.) 

5. o Have not used the WWC to inform any decisions (Go to 

question 12.) 

6. O Don't know (Go to question 12.) 
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13. To what extent, if at all, have you or your staff used information in 
the WWC to do any of the following? 

(Please choose one response for each action.) 

To a 







To a very To a large 
large extent extent 


moderate 

extent 


To a small 
extent 


Not at all 


Don't know 


13a. 


Inform professional development of teachers 


O 


o 


O 


O 


O 


O 


13b. 


Advise s that are not makino AYR on 

potential Interventions 


o 


o 


O 


O 


o 


o 


13c. 


Deveiop improvement plans 


o 


o 


o 


o 


o 


o 


13d. 


Inform curriculum decisions 


o 


o 


o 


o 


o 


o 


13e. 


Other use (Please specify below) 


o 


o 


o 


o 


o 


o 



Other use: 



14. To what extent have you or your staff used the WWC’s Intervention 
Reports to inform decisions on effective education practices? 

Intervention Reports provide an assessment of the efficacy of 
interventions based on existing research that meets certain 
standards. 

(Check only one answer) 

1 . o Very large extent 

2. O Large extent 

3. O Moderate extent 

4. O Small extent 

5. o Don't know 

6. o Have not used the WWC Intervention Reports to Inform any 

decisions (Go to question 18.) 
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18. To what extent have you or your staff used the WWC’s Practice 
Guides to inform decisions on effective education practices? 

Practice Guides are deveioped by a panei of experts and provide 
recommendations to heip educators address common ciassroom or 
schooi-wide chaiienges. 

(Check only one answer) 



1- o 


Very iarge extent 


2. O 


Large extent 


3. O 


Moderate extent 


4. O 


Small extent 


5. O 


Don't know 



6. O Have not used the WWC Practice Guides to inform any 
decisions (Go to question 22.1 

22. To what extent have you or your staff used the WWC’s Quick 
Reviews to inform decisions on effective education practices? 

Quick Reviews are designed to heip educators and poiicy makers 
assess the quaiity of recentiy reieased research papers and reports. 

(Check only one answer) 

1 . o Very large extent 

2. O Large extent 

3. O Moderate extent 

4. O Small extent 

5. O Don't know 

6. O Have not used the WWC Quick Reviews to inform any 

decisions (Go to question 25.) 
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25. How likely or unlikely would you or your staff be to increase your 
usage of the WWC if any of the following information were added to 
the website? 

(Please choose one response for each type of information.) 

Neither 









Somewhat 


likely nor 


Somewhat 


Very 








Very Likely 


Likely 


unlikely 


unlikely 


unlikely 


No opinion 


25a. 


Additionai topic areas 


O 


O 


O 


O 


O 


O 


25b. 


Additionai practice guides 


O 


o 


O 


O 


O 


O 


25c. 


Additionai studies reviewed 


o 


o 


o 


o 


O 


o 


25d. 


Additionai information on interventions based 
on studies that may not meet WWC standards 


o 


o 


o 


o 


O 


o 


25e. 


A broader definition of what studies meet 

WWC standards 


o 


o 


o 


o 


O 


o 


25f. Additionai practices with positive reviews 


o 


o 


o 


o 


O 


o 


25g. 


Reviews of efficacy of programs being used 
or considered in my 


o 


o 


o 


o 


O 


o 


25h. 


Reviews that are more timely 


o 


o 


o 


o 


O 


o 


25i. 


A greater number of intervention reports 
showing positive effects 


o 


o 


o 


o 


O 


o 


25j. 


An easier website to navigate 


o 


o 


o 


o 


O 


o 


25k. 


Other information (Please specify below) 


o 


o 


o 


o 


O 


o 



Other information: 



Source: GAO survey of states’ and school districts’ use of educational clearinghouses. 
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The following tables contain the estimates and associated confidence 
intervals for the data displayed in figures 8, 10, 12, and 14. 



Table 5: Estimates and Confidence Intervals for Figure 8 

Q8. From which of the following source(s) did you or your staff hear about the WWC? 


Label 


Response 


Percentage 


Lower bound 


Upper bound 


Conferences 


Yes 


72.72 


65.26 


80.19 


Conferences 


No 


21.83 


15.17 


29.77 


Conferences 


Don’t know 


5.45 


2.36 


10.50 


Peers 


Yes 


66.43 


58.39 


74.47 


Peers 


No 


27.80 


20.22 


35.37 


Peers 


Don’t know 


5.77 


2.32 


11.62 


Regional Education Labs 


Yes 


47.14 


38.40 


55.89 


Regional Education Labs 


No 


45.24 


36.51 


53.97 


Regional Education Labs 


Don’t know 


7.62 


3.81 


13.34 


U.S. Department of Education 


Yes 


71.93 


64.41 


79.46 


U.S. Department of Education 


No 


22.85 


16.17 


30.72 


U.S. Department of Education 


Don’t know 


5.22 


2.00 


10.80 



Source: GAO analysis. 



Table 6: Estimates and Confidence Intervals for Figure 10 

Q13: To what extent, if at all, have you or your staff used information in the WWC to do any of the following? 



Label 


Response 


Percentage 


Lower bound 


Upper bound 


Inform professional development of teachers 


To a very large extent 


12.33 


6.24 


21.15 




To a large extent 


19.49 


11.74 


29.45 




To a moderate extent 


38.12 


28.25 


48.00 




To a small extent 


19.48 


11.77 


29.35 




Not at all 


5.90 


2.40 


11.80 




Don’t know 


4.67 


1.09 


12.44 


Advise schools that are not making adequate 


To a very large extent 


13.23 


6.90 


22.23 


yearly progress on potential interventions 


To a large extent 


10.12 


5.12 


17.48 




To a moderate extent 


28.21 


18.82 


39.24 




To a small extent 


11.42 


5.69 


19.82 




Not at all 


28.65 


19.55 


39.23 




Don’t know 


8.36 


3.54 


16.16 


Develop school improvement plans 


To a very large extent 


12.07 


6.19 


20.56 




To a large extent 


14.79 


7.59 


24.99 




To a moderate extent 


34.83 


24.68 


44.98 
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Label 


Response 


Percentage 


Lower bound 


Upper bound 




To a small extent 


19.15 


11.72 


28.63 




Not at all 


13.37 


7.35 


21.73 




Don’t know 


5.79 


1.80 


13.36 


Inform curriculum decisions 


To a very large extent 


14.95 


8.53 


23.61 




To a large extent 


21.99 


14.15 


31.65 




To a moderate extent 


36.50 


26.71 


46.28 




To a small extent 


19.50 


12.02 


29.00 




Not at all 


3.31 


0.92 


8.23 




Don’t know 


3.75 


0.63 


11.47 



Source: GAO analysis. 



Table 7: Estimates and Confidence Intervals for Figure 12 

Q14: To what extent have you or your staff used the WWCs intervention reports to inform decisions on effective education practices? 

Q18: To what extent have you or your staff used the WWCs practice guides to inform decisions on effective education practices? 

Q22: To what extent have you or your staff used the WWCs quick reviews to inform decisions on effective education practices? 


Label 


Response 


Percentage 


Lower bound Upper bound 


Intervention reports 


Very large or large extent 


20.98 


13.66 


29.99 




Moderate extent 


29.42 


20.13 


38.71 




Small extent 


23.99 


15.26 


34.67 




Don’t know 


4.60 


1.48 


10.51 




Have not used the WWC intervention report to 
inform decisions 


21.01 


13.23 


30.73 


Practice guides 


Very large or large extent 


10.06 


5.28 


16.95 




Moderate extent 


27.78 


18.84 


38.23 




Small extent 


16.77 


10.12 


25.44 




Don’t know 


2.91 


0.70 


7.75 




Have not used the WWC practice guides to inform 
decisions 


42.48 


32.26 


52.70 


Quick reviews 


Very large or large extent 


12.93 


6.66 


21.92 




Moderate extent 


16.66 


9.79 


25.70 




Small extent 


24.65 


15.93 


35.21 




Don’t know 


5.78 


2.35 


11.55 




Have not used the WWC quick reviews to inform 
decisions 


39.98 


30.16 


49.81 



Source: GAO analysis. 
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Table 8: Estimates and Confidence Intervals for Figure 14 

Q25: How likely or unlikely would you or your staff be to increase your usage of the WWC if any of the following information were added 
to the Web site? 



Label 


Response 


Percentage 


Lower 

bound 


Upper 

bound 


Reviews of efficacy of programs being used or considered in 
my district 


Very likely or somewhat likely 


67.92 


59.77 


76.06 


A greater number of intervention reports showing positve 
effects 


Very likely or somewhat likely 


66.54 


58.30 


74.77 


Additonal practices with positives reviews 


Very likely or somewhat likely 


60.49 


51.95 


69.03 


Additional Topic Areas 


Very likely or somewhat likely 


57.27 


48.66 


65.88 


Additional information on interventions based on studies that 
may not meet WWC standards 


Very likely or somewhat likely 


55.97 


47.43 


64.51 


Additional Practice Guides 


Very likely or somewhat likely 


55.80 


47.12 


64.47 


Additional studies reviewed 


Very likely or somewhat likely 


54.89 


46.24 


63.53 


Reviews that are more timely 


Very likely or somewhat likely 


49.95 


41.32 


58.57 


A broader definition of what studies meet WWC standards 


Very likely or somewhat likely 


47.38 


38.70 


56.05 


An easier web site to navigate 


Very likely or somewhat likely 


47.01 


38.30 


55.73 


Other Information 


Very likely or somewhat likely 


17.37 


8.69 


29.58 


Source: GAO analysis. 



Information from in addition to interviews with teacher, principal, and research organizations, 

Teachers, Principals, and obtained information about the usefulness of the WWC by administering 

Researchers ^ questiormaire at four conferences of teachers and principals. Table 9 

provides more information about the conferences we attended. 



Table 9: Conferences Attended to Administer Questionnaires to Teachers and Principals 


Conference 


Regional/ national 


Location 


Conference dates 


Attendance dates 


National Council of Teachers of 
Mathematics 


Regional 


Nashville, Tennessee 


November 18-20, 
2009 


November 19-20, 2009 


National Council of Teachers of 
English 


National 


Philadelphia, 

Pennsylvania 


November 19-24, 
2009 


November 21 -22, 2009 


ASCD (formerly the Association of 
Supervisors and Curriculum 
Developers) 


National 


San Antonio, Texas 


March 6-8, 2010 


March 7-8, 2010 


National Association of Secondary 
School Principals 


National 


Phoenix, Arizona 


March 12-14, 2010 


March 12-13, 2010 



Source: GAO. 
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We selected these conferences because they were relevant to segments of 
the WWC’s target population that we were not reaching through our 
survey and they were held at times that coincided with our report time 
frames. At each of these conferences, conference organizers agreed to 
have GAO have a table either inside the exhibit hall or just outside it. The 
questiormaires included questions on awareness and use of WWC — 
including use of specific products and use of other information sources to 
identify effective educational practices. For those who had not used the 
WWC, we also asked them to specify the reason they had not used it. The 
information gathered through the questiormaires is not generalizable and 
does not represent the views of teachers and principals nationwide. 
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Estimated percent of 
district officiais that find 


Estimated percent of 
district officiais that find 


Estimated percent of 
district officiais find this 






this source Very Useful 


this source Somewhat or 


source not at aii useful 




Source of information 


or Useful 


Siightiy Usefui 


or no opinion 


Not used 


Personal Experience 


80.6 


16.6 


2.3 


0.5 




(76.46, 84.79) 


(12.73, 20.53) 


(0.90, 4.67) 


(0.06, 1.67) 


Local data and/or internal 


77.4 


14.9 


6.2 


1.5 


research 


(72.85, 81 .84) 


(11.30, 19.13) 


(3.79, 9.58) 


(0.48, 3.55) 


Peer Conferences 


77.3 


16.7 


4.2 


1.7 




(72.93, 81.74) 


(12.88, 20.58) 


(2.28, 7.14) 


(0.60, 3.73) 


Mentors/Colleagues 


74.5 

(69.97, 79.08) 


21 

(16.74, 25.17) 


3.2 

(1.53, 5.84) 


1.3 

(0.42, 3.05) 


Education-related periodicals 


70.9 


23 


3.2 


2.9 




(66.00, 75.72) 


(18.50, 27.50) 


(1.48, 6.02) 


(1.34, 5.49) 


Associations of educators or 


65 


23.1 


6.3 


5.6 


researchers 


(59.94, 70.04) 


(18.59, 27.57) 


(3.83, 9.66) 


(3.39, 8.72) 


Academic journals 


64.3 


27.7 


4.7 


3.3 




(59.22, 69.41) 


(22.90, 32.45) 


(2.59, 7.81) 


(1.64, 5.85) 


Online databases (ERIC or 


54.4 


30.6 


7.9 


7.1 


others) 


(49.10, 59.63) 


(25.62, 35.53) 


(5.12, 11.59) 


(4.58, 10.52) 


Community and parent input 


51.7 

(46.48, 56.97) 


39.9 

(34.67, 45.06) 


6.2 

(3.82, 9.53) 


2.2 

(0.95, 4.19) 


University based research 


50.3 


35.4 


9.3 


5 


institutions 


(45.11, 55.49) 


(30.25, 40.45) 


(6.30, 13.16) 


(2.94, 7.98) 


Regional Educational 


45.9 


31.3 


11.5 


11.4 


Laboratories 


(40.66, 51 .03) 


(26.32, 36.18) 


(8.18, 15.59) 


(8.16, 15.36) 


State government offices 


39.8 


37.8 


15.79 


6.7 


and/or outreach centers 


(34.68, 45.01) 


(32.64, 42.88) 


(11.87, 20.10) 


(4.33, 9.91) 


Other federal outreach centers 


17.9 


34.7 


24 


23.3 




(13.99, 21.83) 


(29.65, 39.77) 


(19.41, 28.67) 


(18.82, 27.86) 


What Works Clearinghouse 


24.4 


14.8 


16.1 


44.7 




(20.07 28.76 


(11.20, 19.09) 


(12.25 20.57) 


(39.37 49.96) 


Doing What Works 


22.8 


13.8 


14.2 


49.3 




(18.40, 27.10) 


(10.42, 17.68) 


(10.52, 18.50) 


(44.03, 54.62) 


Non-profit organization 


21.7 


43.9 


17 


17.4 




(17.63, 25.83) 


(38.58, 49.11) 


(13.06, 21.60) 


(13.22, 21.59) 


RANDs Promising Practices 


10.1 


14 


18.9 


57.1 




(7.34, 13.36) 


(10.48, 18.09) 


(14.54, 23.25) 


(51.77, 62.40) 


Child Trends 


9.7 


14.5 


16.73 


59.1 




(6.74, 13.29) 


(10.93, 18.77) 


(12.84, 21.25) 


(53.84, 64.33) 



Source: GAO analysis of survey results from the following questions: (4) How useful, if at all, are each of the following research 
evaluation resources to you or your staff in identifying effective practices to implement in your district?; and (5) Recognizing that 
research evaluation resources are not necessarily the primary sources of information used to identify effective education practices, 
GAO is also interested in the role of other information sources. How useful, if at all, are each of the following sources to you or your 
staff in identifying effective practices to implement in your district? 
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Appendix III: lES and WWC Response to 
Expert Panel Recommendations 



Expert panel recommendation lES/WWC response Implementation status 

1. Full review. lES should lES is considering an appropriate mechanism and time for Under consideration 

commission a full review of the conducting a complete review of the WWC. lES beiieves that the 

WWC, including a review of the first 2 years of the current contract necessitated a tremendous 

Clearinghouse’s mission and of development effort to transfer the infrastructure of the 

the WWC practice guides, Clearinghouse in year one from one contractor to another, and in 

which the panel did not attempt year two, to complete reviews in a consistent manner that began 

to evaluate. The panel also under the original contract. Now that the Clearinghouse is more 

recommends that lES consider clearly in the production phase, this may be the appropriate time to 

instituting a regular review plan for a complete review. 

process to ensure that the 

WWC Is using the most 

appropriate standards In its 

work. 



2(i). Protocol templates. WWC 

should develop standards for 
crossover and assignment 
noncompliance, and for 
adjusting intention to treat 
effects across studies. 



lES is currently considering having the WWC develop a standard 
for assessing crossover compliance, following the process recently 
used to revise its attrition standard. 

Currently, the WWC documents crossover reported in studies. 
Principal investigators have discretion to use this information to 
determine whether a study represents a reasonable test of the 
intervention. Evidence of crossover and assignment 
noncompliance is documented in the intervention report and its 
appendix table A.1 . Readers can use that information to assess 
the findings. lES agrees there is value in adjusting intent-to-treat 
effects for compliance, but believes this adjustment is inconsistent 
with its goal of having the WWC be transparent in how it reports 
findings. Making its own estimates to account for compliance will 
lead to differences between what the WWC reports and what is 
found in publicly available literature. 



Under consideration 



Currently the WWC does adjust for clustering when authors report 
their findings incorrectly. However, the purpose of the clustering 
adjustment is to correct for an analytic problem In the methods 
authors use to estimate variances, which generally causes them to 
overstate the precision of their findings. In contrast, adjusting for 
compliance will yield an alternate estimate of effects that may differ 
from the one reported by the study. 
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Expert Panel Recommendations 



Expert panel recommendation 


lES/WWC response 


Implementation status 


2(ii). Protocol templates. 

Develop standards for 
documenting the program 
received in the control arm of 
randomized experiments (or by 
members of the comparison 
group in quasi-experimental 
designs), and potentially 
incorporating this information in 
making comparisons across 
studies and/or interventions. 


Though not based on a standard, WWC practice is for reviewers to 
document the counterfactual in study review guides and in 
intervention reports (the information is reported in appendix table 
A.1). Reviewers routinely send author queries for this information, 
if it is not provided in the study. 

lES has asked the WWC to assess how other review organizations 
report counterfactual information and the utility of incorporating this 
information into its reports. lES officials are also considering an 
alternative approach that would code information about the 
counterfactual in a study into the study database, which then 
would generate summary tables that would report results for 
studies that have similar counterfactuals. This approach has 
downsides as well, since the set of counterfactuals could be quite 
varied and many assumptions would have to be made to group 
counterfactuals together. We are therefore proceeding cautiously 
in making any changes to current WWC practice. 


Considered but not 
planning to implement 


2 (ill). Protocol templates. 

Revise standards for multiple 
comparisons in light of the 
recent research report by Peter 
Schochet entitled Guidelines for 
Multiple Testing in Experimental 
Evaluations of Educational 
Intervention.. 


WWC staff consulted with Dr. Schochet to Investigate the 
possibility of revising the multiple comparison standards. 

Dr. Schochet indicated that his report focused on issues related to 
multiple comparisons within single studies. It did not tackle issues 
related to multiple comparisons Issues that may arise when 
synthesizing evidence for a set of studies. WWC procedures are 
consistent with his report for handling multiple comparisons within 
a study. 


Considered but not 
planning to implement 


2(iv). Protocol templates. 

Reconsider the current process 
of setting different attrition 
standards in different topic 
areas. 


At the time of the National Board of Education Sciences Expert 
Panel’s data collection, the WWC was already reviewing its 
attrition standards. The WWC released new attrition standards in 
December 2008 in the Procedures and Standards Handbook. The 
new standards requires a principal Investigator in a topic area to 
choose one of two well-specified attrition boundaries, and the 
standards include guidance on how to choose between the 
boundaries based on the nature of research in the topic area. 

The attrition discussion is in the WWC Procedures and Standards 
Handbook (Version 2.0) posted on the Clearinghouse’s Web site. 


Implemented 
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Expert Panel Recommendations 



Expert panel recommendation 


lES/WWC response 


Implementation status 


2(v). Protocol templates. 

Establish a protocol to keep 
track of potential conflicts of 
interest, such as cases where a 
study is funded or conducted by 
a program developer, and 
consider making that 
information available in its 
reports. 


lES is considering options for collecting and documenting potential 
conflicts of interest. 

Sources of funding are rarely Included In published documents 
beyond government and foundation support. An alternate source 
of Information for tracking potential conflicts of interest would be for 
the WWC to request that study authors Identify their source of 
funding, which would provide the WWC with a basis for flagging a 
potential conflict of interest. Any effort would depend on 
cooperation from authors because the WWC has no leverage to 
formally require authors to declare potential conflicts (which some 
academic journals require as a condition for publication). WWC’s 
experience to date is that study authors frequently fail to respond 
to requests for additional information, and lES officials expect that 
many study authors likewise will not respond to requests for 
information about funding sources, or may judge that it is not in 
their proprietary interest to provide the information. Currently the 
WWC only queries authors in cases where the Clearinghouse 
needs additional information. Querying all authors and tracking 
their responses would increase costs for intervention reports. 


Under consideration 




Another potential option is to ask developers, when they are 
reviewing the list of studies WWC found during the literature 
search for comprehensiveness, to note any studies that they 
funded. 




2(vi). Protocol templates. 

Define precisely the standards 
for “randomization” in a 
multilevel setting. 


The current version of the handbook gives guidance on standards 
for random assignment in simple cases. The next version of the 
handbook (forthcoming in 2010) will provide guidance and 
examples for multilevel settings, with explicit guidance on 
acceptable practice and potential issues with random assignment 
in a multilevel setting. 


Implemented 


3. Documentation of search 
process. WWC should expand 
the protocol templates to specify 
more explicit documentation of 
the actual search process used 
in each topic area and maintain 
a record of the results of the 
process that can be used to 
guide decision making on future 
modifications. 


lES asked the WWC to review the search process. The 

WWC now takes steps to ensure that search records are 
maintained. Each team and the library maintain a record of 
conducted searches. More documentation on the process will be 
included in the forthcoming revision of the handbook. 


Implemented 
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Expert Panel Recommendations 



Expert panel recommendation 


lES/WWC response 


Implementation status 


4. Reliability of eligibility 
screening. WWC should 
conduct regular studies of the 
reliability of the eligibility 
screening process, using two 
independent screeners, and use 
the results from these studies to 
refine the eligibility screening 
rules and screening practices. 


The WWC is undertaking a pilot using five recent evidence reports 
in different topic areas. Because WWC screeners are encouraged 
to pass to the next stage any study for which they are uncertain 
about eligibility, the proportion of eligible studies that are excluded 
is the salient error rate (the other source of error is when screeners 
include an ineligible study in a review, but this error is then offset 
by the review). 

lES officials are not aware of any established standards for 
acceptable error rates (there are tradeoffs between making Type 1 
vs. Type II errors relating to cost), but will examine this issue 
further. If the screening error rate is larger than the lES and the 
WWC believes is acceptable, lES officials will assess whether 
additional training or two screeners is appropriate given the 
different costs and benefits of each approach. 


Under consideration 


5. Documentation of 
screening process. WWC 

reports should include a flow 
chart documenting the flow of 
studies through each review 
and number of studies excluded 
at each point, and a table of 
excluded studies, listing specific 
reasons for exclusion for each 
study. 


Currently, reference lists for WWC intervention reports include all 
studies, both eligible and ineligible, located in the search process. 
Ineligible studies are flagged with the primary reason for not 
qualifying for further WWC review. Intervention reports do not list 
materials such as product descriptions or reviews of products that 
are deemed not relevant to the intervention being reviewed. 

To make the number of studies (both eligible and ineligible studies) 
more apparent to readers, the WWC will add a text box to 
intervention reports located in front of the iisting of reports. The 
text box will summarize the number of studies that met different 
conditions (this approach currently is used for reports in which 
none of the studies meet standards). The box will serve the same 
purpose as a flow chart but the codes used to describe the final 
status for reports will be the same ones currently used in the 
citation appendix. The WWC plans to begin including the text box 
in reports released in 2010 and thereafter. 


Implemented 


6. Misalignment adjustment. 

In cases where a study analysis 
is “misaligned,” WWC staff 
should request that study 
authors reanalyze their data 
correctly, taking into account the 
unit of randomization and 
clustering. The panel 
recommends that the results 
from the process be compared 
to the adjustment procedure 
currently specified, to develop 
evidence on the validity of the 
latter. 


Ideally, the primary source for reanalyses of data would be study 
authors. However, as noted above in response to recommendation 
2(v), it is common for authors not to respond to the WWC’s 
requests for additional information. Reanalyzing the data also 
would require additional effort by the authors and would run into 
difficulties when studies are dated or are based on data that has 
been destroyed to comply with confidentiality or privacy 
restrictions. 

The WWC recently undertook a survey of published clustering 
estimates. It found that the WWC’s current default clustering 
correction is consistent with published estimates for achievement 
and behavioral outcomes. The WWC will continue to monitor 
research developments on this topic. 


Considered but not 
planning to implement 
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Expert Panel Recommendations 



Expert panel recommendation 


lES/WWC response 


Implementation status 


7. Combining evidence 
across multiple studies. WWC 

should re-evaluate its 
procedures for combining 
evidence across studies, with 
specific attention to the issue of 
how the rules for combining 
evidence can be optimally 
tuned, given the objectives of 
the WWC review process and 
the sample sizes in typical 
studies for a topic area. 


There are, of course, many possible ways to summarize evidence. 
Given its intended broad and primarily nontechnical audience, the 
WWC’s current approach is designed to be transparent and easily 
explained. lES believes that having the WWC conduct its own 
analyses to estimate intervention effects, as statistical meta- 
analyses do, would be Inconsistent with these goals. 

However, as an alternative to modifying the WWC’s main approach 
for reporting findings, lES is considering having the WWC conduct 
supplemental meta-analyses related to specific questions of 
interest, and releasing these findings as a separate report that 
would complement Intervention reports. For example, a report 
could analyze whether computer mediated approaches to teaching 
reading are more or less effective than approaches that rely solely 
on teachers, based on already-released interventions reports. 
Having a separate report enables the WWC to continue using its 
current transparent approach, while also using statistical 
techniques that combine evidence in other ways. 


Under consideration 


8. Reporting, (i) Published 
reports on the Web site should 
include the topic area protocols, 
as well as more information on 
the screening process results 
that led to the set of eligible 
studies actually summarized in 
the topic area reports, (ii) WWC 
should make available its 
Standards and Procedures 
Handbook, including 
appendixes, as well as all other 
relevant documents that 
establish and document its 
policies and procedures. 


Topic area protocols are available on the topic area home pages. 
Just after the expert panel’s report, the WWC released its 
Procedures and Standards Handbook in December 2008. A 
revision currently Is under way that will include more detail on the 
screening process. See the response to (3) above related to 
results of the screening process and the response to (5) regarding 
the results of the screening process. 


Implemented 


9. Practice guides. Clearly 
separate practice guides from 
the topic and intervention 
reports. 


lES agrees that these products need to remain distinct. Practice 
guides are on a separate Web site tab that separates them from 
intervention reports. The next revision of the handbook 
(forthcoming in 2010) will include a chapter describing the practice 
guide development process and how It Is different from that of the 
evidence reports. 

The recently released guide on What Works for Practitioners also 
provides more information on reports and practice guides, and the 
WWC is preparing a video tutorial that will explain the differences 
to users. 


Implemented 
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Expert Panel Recommendations 



Expert panel recommendation lES/WWC response 



Implementation status 



10. Outreach and 
collaboration with other 
organizations, (i) The WWC 

should build and maintain a 
relationship with nationai and 
internationai organizations 
focusing on systematic reviews 
to engage in the broader 
scientific community and iearn 
about the latest standards and 
practices, (ii) The WWC shouid 
convene working groups with a 
mixture of researchers 
(including specialists in 
education research and 
systematic reviews) to address 
the development of new 
standards for the review and 
synthesis of studies. 



The WWC tries to keep abreast of deveiopments in the field, for Implemented 
example, by routinely checking materials from the Cochrane 
Collaborative when developing new standards or approaches. 

Most recently, the WWC has undertaken the following outreach 
efforts to connect with other organizations conducting systematic 
reviews: 

• The WWC sponsored a forum on research methods in 
December 2008 that featured speakers from the National 
Academy of Sciences, the National Cancer Institute, and the 
Cochrane Collaboration. 

• In June 2009 WWC staff attended the Cochrane conference on 
practice guides in June to learn about state of the art methods in 
research synthesis and practice guides. 

• The WWC is presenting a workshop on WWC standards at the 
upcoming annual conference of the Association of Public Policy 
and Management. 

The WWC has also met with six international contacts (from 
Sweden, Denmark, Hungary, England, Interamerican Development 
Bank, and Trinidad/Tobago) in response to inquiries about how 
governments or organizations could implement their own 
clearinghouse operations. 

Recently the WWC began a webinar series to disseminate its new 
practice guides. The webinar includes researchers and 
practitioners in its audience. 

The WWC convened two groups of researchers to develop its 
forthcoming standards on single-subject designs and regression 
discontinuity designs. It will continue to bring together researchers 
as needs for new standards are identified. This approach will 
continue to be used for developing new standards. 



Source: GAO analysis of lES and WWC data. 
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Appendix IV: Comments from the 
Department of Education 



UNITED STATES DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 

THE DIRECTOR 

July 6, 2010 



Ms. Cornelia M. Ashby 
Director, Education, Workforce, 
and Income Security Issues 
U.S. Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 

Dear Ms. Ashby: 

Thank you for the opportunity to respond to the Government Accountability Office (GAO) draft report, 
Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works 
Clearinghouse (GAO- 10-644). 

This report indicates that GAO, as well as a congressionally mandated panel of experts, found that the 
What Works Clearinghouse’s review process follows accepted standards for evaluating research on the 
effectiveness of education interventions. It notes that the Clearinghouse is responding to 
recommendations made by the expert panel to further improve its review and reporting processes, and to 
researchers who have criticized the WWC for excluding some types of research designs that may be 
appropriate for evaluating certain education programs, such as special education. It also recognized that 
product output and scope of the WWC have increased. 

While the report emphasized the need for more timely release of WWC reports, it noted, but did not 
highlight, other important findings from the sehool district survey that point to important avenues for 
improvement. For example, a higher proportion of district respondents cited having evidence reviews of 
programs that they might use in their locd areas (68 percent) or that show positive effects (67 percent) as 
a reason to use WWC products than cited getting reviews more quickly (50 percent) (Figure 1 5, page 36). 
Given that the timetable of WWC reports is not posted publicly, we are not sure how to interpret districts’ 
concerns with timeliness, but we have heard their need for more relevant evidence reports covering more 
educational areas and programs. As GAO noted, a backlog of reports that built up during a time of senior 
leadership turnover has been resolved, and many new reports are soon to be released. In addition, the 
Institute of Education Sciences is continuing its efforts to ensure that expansion into new topic areas is 
responsive to the needs of the field. 



555 NEW JERSEY AVH, NW. WASHINGTON, D.C 20208 
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We appreciate the opportunity to comment on the draft report. The Department of Education has 
prepared the attached comments in response to your draft report. If you have any questions regarding this 
response, please contact Dr. Rebecca Maynard, Commissioner for Education Evaluation and Regional 
Assistance, in the Institute of Education Sciences at (202) 208-1289. 



Sincerely, 






Jetm Q. Easton 
Director 



Attachments 
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Attachment 1 

lES Response to GAO Report: Improved Dissemination and Timely Product Release Would 
Enhance the Usefulness of the What Works Clearinghouse, Draft 



Recommendation 1: 

To consistently release WWC products in a timely manner, we recommend the Secretary of Education 
direct lES to develop and implement strategies that help avoid future backlogs and ensure that lES' 
review and approval processes keep pace with increased contractor production. Strategies could include 
shifting lES resources to ensure sufficient staff time for managing the peer review process and 
streamlining its approval process. 

Response: 

The Department of Education (Department) agrees with GAO’s recommendation and is developing and 
implementing strategies to ensure that lES’ review and approval processes keep pace with production. 
The Director of the Institute of Education Sciences has recently appointed a Commissioner of the 
National Center for Education Evaluation and Regional Assistance who will oversee the content and 
operations of the What Works Clearinghouse. 

Recommendation 2: 

To better track the usefulness and costs of the WWC, we recommend that the Secretary of Education 
direct lES to: 

• Incorporate findings from its cost studies to develop performance measures related to costs, such 
as identifying a range of acceptable costs per product and using that information to monitor 
contractor spending. 

• Develop performance measures related to product usefulness and periodically assess whether 
WWC products are meeting the needs of target audiences, by gathering information on product 
usefulness in the proposed survey or through other means. 

Response: 

First, as noted in the report, WWC is currently conducting assessments for streamlining procedures and 
costs at lES’ request. The results, due at the end of October 2010, will be incorporated into future WWC 
work plans and lES’s monitoring of contractor performance and costs. lES intends to undertake such 
assessments on an ongoing basis for continuous improvement. 

Second, the WWC will continue to monitor and evaluate consumer use and assessment of WWC 
products, as well as gather information regarding suggestions for improvements. In addition, the WWC 
will be included as part of lES’ review of its own performance, which will include surveying customers 
on satisfaction. 

In addition, the Clearinghouse already has in place a number of tools for users to provide input, and we 
will continue to emphasize their availability to the education conununity. The WWC Help Desk allows 
users to contact the WWC to send suggestions and ask questions about specific products. Users contact 
the Help Desk through an 800 number or through the WWC Web site. Since 2007, the WWC has 
responded to over 2,100 inquiries to the Help Desk. 
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Users have suggested over 200 interventions for the WWC to review, and have suggested over 30 
different topics for practice guides. This information is continuously reviewed by WWC staff and 
informs priorities. The WWC has used Web-based tools for soliciting specific feedback from users. For 
example, the WWC has posted potential practice guide topics on a Web-based survey, having users 
“vote” for their favorite topics. The forthcoming practice guides on Reading Comprehension, Fractions, 
and Teaching Writing were all identified through this process. 

In 2007, 2008, and 2009, the WWC conducted focus groups to solicit feedback from educators on several 
aspects of its work. These included such topics as the usability of the WWC Web site, which led to 
improvement to the site fimctionality, and the format of practice guides, which led to changes in the 
layout of the guides. 

Visits to the WWC Web site have increased substantially over time. , Between fiscal years 2008 and 2009, 
the number of visits rose by 45 percent to 772,000. And, to date in 2010, there have been 618,000 visits 
to the Web site, an increase of 21 percent relative to the same period in fiscal year 2009. lES will 
continue to track all of these measures to ensure that we are using solid information on customer use and 
satisfaction. 

Recommendation 3: 

To reach more members of the target audience, we recommend the Secretary of Education direct lES to 
assess and improve its dissemination efforts to promote greater awareness and use of the WWC, for 
example, by developing a way to inform school districts of new products or encouraging educator 
professional development programs to focus on research-based practices such as those discussed in 
practice guides. 

Response: 

The Department agrees with GAO’s recommendation and is developing further means to assess and 
improve dissemination efforts. lES’ monitoring of the WWC contract requires implementation of an 
Annual Communication Plan that specifies how the contractor will conduct outreach, dissemination, and 
communication activities that include, for example, the WWCFlash, media and trade organization 
outreach, development of dissemination partnerships with organizations and the education community, 
WWC sponsored events and forums, and targeted product outreach. lES has made all WWC products 
available through the Department’s Educational Resources Information Center (ERIC), an online digital 
library of education research and information, and provides RSS feeds of WWC products to ERIC users. 
The Department’s National Library of Education makes all WWC products, together with all items 
indexed in ERIC, available to Google, MSN, and Yahoo search engines for harvesting and, therefore, 
accessible to Web users worldwide. All WWC products are now available to WorldCat, an international 
bibliographic database that is used by libraries around the world to identify and locate resources. 

lES will continue to develop and implement the Communication Plan vehicle to assess and improve 
dissemination, as well as fully utilize other Department dissemination resources. One example of a new 
vehicle that the Clearinghouse has in the works to expand its dissemination partnerships with 
organizations and the education community is a monthly WWC Newsletter on up-to-date information on 
recently released Clearinghouse products as well as those in line for release in the month ahead. 
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The Government Accountability Office, the audit, evaluation, and 
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constitutional responsibilities and to help improve the performance and 
accoimtability of the federal government for the American people. GAO 
examines the use of public funds; evaluates federal programs and policies; 
and provides analyses, recommendations, and other assistance to help 
Congress make informed oversight, policy, and funding decisions. GAO’s 
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